[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 501399803 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002333] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007561] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009010] pid_max: default: 32768 minimum: 301 [ 0.010123] LSM: Security Framework initializing [ 0.011042] Yama: becoming mindful. [ 0.012027] SELinux: Initializing. [ 0.013052] *** VALIDATE selinux *** [ 0.021738] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025587] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027115] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028096] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029094] *** VALIDATE tmpfs *** [ 0.030400] *** VALIDATE proc *** [ 0.032010] *** VALIDATE cgroup *** [ 0.033006] *** VALIDATE cgroup2 *** [ 0.034228] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035138] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037023] Spectre V2 : User space: Vulnerable [ 0.038006] Speculative Store Bypass: Vulnerable [ 0.041366] debug: unmapping init [mem 0xffffffff9d259000-0xffffffff9d260fff] [ 0.044219] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046084] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047047] ... version: 2 [ 0.048018] ... bit width: 48 [ 0.049019] ... generic registers: 4 [ 0.050017] ... value mask: 0000ffffffffffff [ 0.051019] ... max period: 00007fffffffffff [ 0.052032] ... fixed-purpose events: 3 [ 0.053126] ... event mask: 000000070000000f [ 0.055284] rcu: Hierarchical SRCU implementation. [ 0.058510] smp: Bringing up secondary CPUs ... [ 0.059778] x86: Booting SMP configuration: [ 0.060033] .... node #0, CPUs: #1 #2 #3 [ 0.064595] smp: Brought up 1 node, 4 CPUs [ 0.066027] smpboot: Max logical packages: 1 [ 0.067031] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.145507] node 0 deferred pages initialised in 76ms [ 0.147393] devtmpfs: initialized [ 0.148785] x86/mm: Memory block size: 128MB [ 0.151262] gcov: version magic: 0x41383552 [ 0.154059] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.155119] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.156369] pinctrl core: initialized pinctrl subsystem [ 0.157354] [ 0.157920] ************************************************************* [ 0.158024] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159025] ** ** [ 0.160022] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.161020] ** ** [ 0.162023] ** This means that this kernel is built to expose internal ** [ 0.163022] ** IOMMU data structures, which may compromise security on ** [ 0.164025] ** your system. ** [ 0.165023] ** ** [ 0.166022] ** If you see this message and you are not debugging the ** [ 0.167019] ** kernel, report this immediately to your vendor! ** [ 0.168026] ** ** [ 0.169021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.170017] ************************************************************* [ 0.171926] NET: Registered protocol family 16 [ 0.172557] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.173103] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.174109] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.176082] cpuidle: using governor menu [ 0.178874] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.182624] PCI: Using configuration type 1 for base access [ 0.185095] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.195087] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.196062] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.198092] cryptd: max_cpu_qlen set to 1000 [ 0.200306] ACPI: Added _OSI(Module Device) [ 0.201018] ACPI: Added _OSI(Processor Device) [ 0.202025] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.203016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.207092] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.210585] ACPI: Interpreter enabled [ 0.211092] ACPI: PM: (supports S0 S3 S4 S5) [ 0.212049] ACPI: Using IOAPIC for interrupt routing [ 0.213189] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.219358] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.230095] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.233043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.236028] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.240089] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.245241] acpiphp: Slot [2] registered [ 0.248053] acpiphp: Slot [5] registered [ 0.249173] acpiphp: Slot [6] registered [ 0.251191] acpiphp: Slot [3] registered [ 0.252147] acpiphp: Slot [4] registered [ 0.253073] acpiphp: Slot [7] registered [ 0.254079] acpiphp: Slot [8] registered [ 0.256170] acpiphp: Slot [9] registered [ 0.258093] acpiphp: Slot [10] registered [ 0.259113] acpiphp: Slot [11] registered [ 0.261092] acpiphp: Slot [12] registered [ 0.262126] acpiphp: Slot [13] registered [ 0.263086] acpiphp: Slot [14] registered [ 0.264079] acpiphp: Slot [15] registered [ 0.266102] acpiphp: Slot [16] registered [ 0.268120] acpiphp: Slot [17] registered [ 0.269131] acpiphp: Slot [18] registered [ 0.272117] acpiphp: Slot [19] registered [ 0.273076] acpiphp: Slot [20] registered [ 0.274084] acpiphp: Slot [21] registered [ 0.275093] acpiphp: Slot [22] registered [ 0.277134] acpiphp: Slot [23] registered [ 0.278112] acpiphp: Slot [24] registered [ 0.280188] acpiphp: Slot [25] registered [ 0.281092] acpiphp: Slot [26] registered [ 0.283104] acpiphp: Slot [27] registered [ 0.285085] acpiphp: Slot [28] registered [ 0.286100] acpiphp: Slot [29] registered [ 0.287106] acpiphp: Slot [30] registered [ 0.289088] acpiphp: Slot [31] registered [ 0.291094] PCI host bridge to bus 0000:00 [ 0.293019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.295021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.297014] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.299020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.302027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.307034] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.310262] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.314147] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.317381] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.325949] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.330059] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.334107] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.337019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.340092] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.342678] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.347239] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.350036] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.353836] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.360014] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.369697] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.374011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.380152] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.394012] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.398011] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.408017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.417451] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.425014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.435014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.455013] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.470051] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.472307] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.474328] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.476374] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.479209] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.484034] iommu: Default domain type: Passthrough [ 0.486489] SCSI subsystem initialized [ 0.489192] ACPI: bus type USB registered [ 0.491118] usbcore: registered new interface driver usbfs [ 0.494116] usbcore: registered new interface driver hub [ 0.495063] usbcore: registered new device driver usb [ 0.497132] pps_core: LinuxPPS API ver. 1 registered [ 0.499007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.502080] PTP clock support registered [ 0.506065] EDAC MC: Ver: 3.0.0 [ 0.507515] PCI: Using ACPI for IRQ routing [ 0.510477] NetLabel: Initializing [ 0.511007] NetLabel: domain hash size = 128 [ 0.512006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.514068] NetLabel: unlabeled traffic allowed by default [ 0.516115] vgaarb: loaded [ 0.517303] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.519013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.525421] clocksource: Switched to clocksource kvm-clock [ 0.630046] VFS: Disk quotas dquot_6.6.0 [ 0.631382] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.633435] *** VALIDATE ramfs *** [ 0.634462] *** VALIDATE hugetlbfs *** [ 0.636275] pnp: PnP ACPI init [ 0.638476] pnp: PnP ACPI: found 6 devices [ 0.665116] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.669065] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.671950] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.674424] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.677420] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.679464] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.681692] NET: Registered protocol family 2 [ 0.683577] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.687784] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.690123] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.695859] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.699736] TCP: Hash tables configured (established 65536 bind 65536) [ 0.702948] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.706825] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.709392] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.712432] NET: Registered protocol family 1 [ 0.715186] RPC: Registered named UNIX socket transport module. [ 0.717365] RPC: Registered udp transport module. [ 0.718704] RPC: Registered tcp transport module. [ 0.720441] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.722423] NET: Registered protocol family 44 [ 0.723629] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.725408] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.728401] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.730769] PCI: CLS 0 bytes, default 64 [ 0.732526] Unpacking initramfs... [ 2.173171] debug: unmapping init [mem 0xffff9fd0fcc64000-0xffff9fd0fffcffff] [ 2.177991] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.180782] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.184258] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.695564] Initialise system trusted keyrings [ 2.698530] Key type blacklist registered [ 2.703971] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.718279] zbud: loaded [ 2.721527] *** VALIDATE nfs *** [ 2.722888] *** VALIDATE nfs4 *** [ 2.725371] pstore: using deflate compression [ 2.729616] Platform Keyring initialized [ 2.852077] NET: Registered protocol family 38 [ 2.853676] Key type asymmetric registered [ 2.855156] Asymmetric key parser 'x509' registered [ 2.856981] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.859691] io scheduler mq-deadline registered [ 2.861230] io scheduler kyber registered [ 2.862705] io scheduler bfq registered [ 2.864729] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.867644] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.871835] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.874910] ACPI: Power Button [PWRF] [ 2.883714] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.894381] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.914611] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.946031] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.977339] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.983332] Non-volatile memory driver v1.3 [ 2.985414] Linux agpgart interface v0.103 [ 3.026289] virtio_blk virtio1: [vda] 146008 512-byte logical blocks (74.8 MB/71.3 MiB) [ 3.032356] vda: detected capacity change from 0 to 74756096 [ 3.052418] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.056268] vdb: detected capacity change from 0 to 1073741824 [ 3.070916] libphy: Fixed MDIO Bus: probed [ 3.079965] usbcore: registered new interface driver usbserial_generic [ 3.083374] usbserial: USB Serial support registered for generic [ 3.086531] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.092689] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.095388] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.098528] mousedev: PS/2 mouse device common for all mice [ 3.103653] rtc_cmos 00:05: RTC can wake from S4 [ 3.108671] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.109116] rtc_cmos 00:05: registered as rtc0 [ 3.114910] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.116440] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.118189] intel_pstate: CPU model not supported [ 3.123863] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.128844] hid: raw HID events driver (C) Jiri Kosina [ 3.130821] usbcore: registered new interface driver usbhid [ 3.133241] usbhid: USB HID core driver [ 3.134745] drop_monitor: Initializing network drop monitor service [ 3.137538] Initializing XFRM netlink socket [ 3.141342] NET: Registered protocol family 10 [ 3.145291] Segment Routing with IPv6 [ 3.147116] NET: Registered protocol family 17 [ 3.149898] mpls_gso: MPLS GSO support [ 3.156267] RAS: Correctable Errors collector initialized. [ 3.158766] AVX version of gcm_enc/dec engaged. [ 3.160848] AES CTR mode by8 optimization enabled [ 3.246844] sched_clock: Marking stable (3246766563, 0)->(4190732876, -943966313) [ 3.251272] registered taskstats version 1 [ 3.254166] Loading compiled-in X.509 certificates [ 3.256836] zswap: loaded using pool lzo/zbud [ 3.287445] Key type big_key registered [ 3.304388] Key type encrypted registered [ 3.306354] ima: No TPM chip found, activating TPM-bypass! [ 3.308944] ima: Allocated hash algorithm: sha1 [ 3.311542] ima: No architecture policies found [ 3.313661] evm: Initialising EVM extended attributes: [ 3.315931] evm: security.selinux [ 3.318120] evm: security.ima [ 3.319501] evm: security.capability [ 3.321130] evm: HMAC attrs: 0x1 [ 3.323657] rtc_cmos 00:05: setting system clock to 2026-08-19 04:50:43 UTC (1787115043) [ 3.331103] debug: unmapping init [mem 0xffffffff9e203000-0xffffffff9e3fffff] [ 3.334656] debug: unmapping init [mem 0xffffffff9cf82000-0xffffffff9d258fff] [ 3.344138] Write protecting the kernel read-only data: 28672k [ 3.348640] debug: unmapping init [mem 0xffffffff9b603000-0xffffffff9b7fffff] [ 3.351627] debug: unmapping init [mem 0xffffffff9bf14000-0xffffffff9bffffff] [ 3.412311] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.423925] systemd[1]: Detected virtualization kvm. [ 3.426097] systemd[1]: Detected architecture x86-64. [ 3.428584] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.453709] systemd[1]: No hostname configured. [ 3.456194] systemd[1]: Set hostname to . [ 3.459128] random: systemd: uninitialized urandom read (16 bytes read) [ 3.462121] systemd[1]: Initializing machine ID from random generator. [ 3.514433] random: ln: uninitialized urandom read (6 bytes read) [ 3.638738] random: systemd: uninitialized urandom read (16 bytes read) [ 3.642551] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.648488] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.654957] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.568432] device-mapper: uevent: version 1.0.3 [ 4.570812] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.427903] virtio_net virtio0 ens2: renamed from eth0 [ 5.449580] random: fast init done [ 5.552949] scsi host0: ata_piix [ 5.580095] scsi host1: ata_piix [ 5.585390] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.587638] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.459340] dracut-initqueue[582]: RTNETLINK answers: File exists [ 9.826641] random: crng init done [ 9.828035] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.868085] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.073419] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.380488] SELinux: Disabled at runtime. [ 12.446200] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.455547] systemd[1]: Detected virtualization kvm. [ 12.457348] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.040514] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.044303] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.049387] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.053607] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.057310] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.066249] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.073663] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 13.199854] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.623500] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.932823] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.940291] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.146302] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.164531] EDAC sbridge: Ver: 1.1.2 [ 15.537330] Key type dns_resolver registered [ 15.868742] NFS: Registering the id_resolver key type [ 15.871299] Key type id_resolver registered [ 15.872782] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Login Service... Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg328-client login: [ 79.851525] libcfs: loading out-of-tree module taints kernel. [ 79.902104] Key type ._llcrypt registered [ 79.904242] Key type .llcrypt registered [ 80.311589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 80.326658] alg: No test for adler32 (adler32-zlib) [ 81.735723] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 82.813075] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 84.703190] Key type lgssc registered [ 86.682900] Lustre: Echo OBD driver; http://www.lustre.org/ [ 152.834018] hrtimer: interrupt took 3211491 ns [ 237.584303] Lustre: Mounted lustre-client [ 243.257565] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 255.627716] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing check_logdir /tmp/testlogs/ [ 259.922944] Lustre: DEBUG MARKER: oleg328-client.virtnet: executing yml_node [ 263.143142] Lustre: lustre-OST0000-osc-ffff9fd142fc4800: disconnect after 23s idle [ 264.649454] Lustre: DEBUG MARKER: Client: 2.17.56.4 [ 267.316464] Lustre: DEBUG MARKER: MDS: 2.17.56.4 [ 270.022139] Lustre: DEBUG MARKER: OSS: 2.17.56.4 [ 271.918369] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Wed Aug 19 00:55:10 EDT 2026 [ 289.359517] Lustre: Unmounted lustre-client [ 322.489636] Key type lgssc unregistered [ 322.939087] LNet: 8399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 322.945398] LNetError: 8399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 322.969643] LNet: Removed LNI 192.168.203.28@tcp [ 323.995444] Key type .llcrypt unregistered [ 324.000240] Key type ._llcrypt unregistered [ 354.644456] Key type ._llcrypt registered [ 354.647308] Key type .llcrypt registered [ 354.968693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 355.015879] alg: No test for adler32 (adler32-zlib) [ 355.952611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 356.028106] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 356.041338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 361.596379] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 361.803868] LNet: 9277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 361.815415] LNetError: 9277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 361.864089] LNet: Removed LNI 192.168.203.28@tcp [ 363.204308] Key type .llcrypt unregistered [ 363.206178] Key type ._llcrypt unregistered [ 371.493116] Key type ._llcrypt registered [ 371.500204] Key type .llcrypt registered [ 373.374287] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 00:56:51 (1787115411) [ 373.970335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 373.993536] alg: No test for adler32 (adler32-zlib) [ 375.156813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.339198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.376217] LNet: 9697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 375.426441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.466147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 375.485164] LNet: 9704:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 385.497895] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 00:57:03 (1787115423) [ 386.144610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 386.207714] LNet: 10283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 386.294785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 386.540495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 394.595245] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 00:57:13 (1787115433) [ 394.867555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 394.895787] LNet: 10869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 394.908620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.982070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 401.571235] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 00:57:20 (1787115440) [ 401.820233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.842377] LNet: 11455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 401.877770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.983428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 408.946140] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 00:57:27 (1787115447) [ 409.219833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 409.276560] LNet: 12040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 409.321920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 409.443752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 410.008543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 418.042541] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 00:57:36 (1787115456) [ 418.225904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 418.262763] LNet: 12628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 418.304183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 418.418039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 419.028095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 419.612377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 427.803856] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 00:57:46 (1787115466) [ 428.094638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.226699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.343573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 436.275026] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 00:57:55 (1787115475) [ 436.610977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 436.657070] LNet: 13813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 436.674267] LNet: 13813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 436.716974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 436.816143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 436.842402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 437.132434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 437.155441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 437.819376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 437.866476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 438.281648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 438.350938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 438.720211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 438.770287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 439.488479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 439.551854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 439.857835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 439.888607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 440.139331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 440.187657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 440.517767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 440.576374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 441.310902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 441.362824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 441.853666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 441.883370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 442.273929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 442.324865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 442.863869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 442.930357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 448.960980] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 00:58:07 (1787115487) [ 449.476227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 449.555770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 449.708323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 449.766307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 456.640471] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 00:58:15 (1787115495) [ 456.813860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 456.878880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 457.010888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 457.053703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 463.951493] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 00:58:23 (1787115503) [ 464.314085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.396601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.499870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 464.569394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 470.216713] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 00:58:29 (1787115509) [ 470.568279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 470.592873] LNet: 16256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 470.597486] LNet: 16256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 470.624488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 470.686512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 470.733160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 476.928555] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 00:58:35 (1787115515) [ 477.226357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.260343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.323223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 477.353756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 483.343921] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 00:58:42 (1787115522) [ 483.494410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 483.534432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 483.599726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 483.651529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 489.381876] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 00:58:48 (1787115528) [ 489.472563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 489.512658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 489.587726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 490.106331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 495.912786] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 00:58:55 (1787115535) [ 496.156994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 496.244734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.485676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 496.676241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 502.956193] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 00:59:01 (1787115541) [ 503.066617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 503.104664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 503.244268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 503.524176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 509.427444] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 00:59:08 (1787115548) [ 509.658201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 509.691159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.155874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 510.422477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 516.564246] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 00:59:15 (1787115555) [ 516.946154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 516.986468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 517.060854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 517.106340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 523.687278] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 00:59:22 (1787115562) [ 524.209581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 524.288323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 524.364024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 524.430542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 532.962565] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 00:59:31 (1787115571) [ 533.245476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 533.316815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 533.492224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 542.349526] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 00:59:40 (1787115580) [ 542.637709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.693418] LNet: 22279:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 542.702023] LNet: 22279:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 542.753948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.834991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 549.079417] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 00:59:48 (1787115588) [ 549.191145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.240247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.319737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 549.694867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 555.432823] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 00:59:54 (1787115594) [ 555.755538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 555.823447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 555.963616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 556.039790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 562.875405] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 01:00:01 (1787115601) [ 563.352663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 563.440597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 563.548556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 563.597760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 570.154720] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 01:00:08 (1787115608) [ 570.214708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 570.247680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 570.312520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 570.355984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 577.118167] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 01:00:15 (1787115615) [ 577.291542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 577.333762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 577.427748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 577.491544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 584.947615] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 01:00:23 (1787115623) [ 585.221829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 585.282885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 585.359331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 585.458376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 585.499077] LNetError: 25811:0:(peer.c:2277:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 585.582405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 585.658912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 592.147851] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 01:00:30 (1787115630) [ 592.246120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.332122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.485403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.ffnCFCpFsP [ 592.586832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.ffnCFCpFsP [ 592.644358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.ffnCFCpFsP [ 599.558365] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 01:00:38 (1787115638) [ 599.987776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 600.049716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 600.140596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 600.198732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 600.261075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 607.811519] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 01:00:46 (1787115646) [ 607.987242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 608.072757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 608.230299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 608.273381] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 610.806616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 613.039031] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 615.159161] Key type lgssc registered [ 616.772130] Lustre: Echo OBD driver; http://www.lustre.org/ [ 620.979252] LNetError: 27573:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.203.28@tcp99 has no available nets [ 623.594463] Key type lgssc unregistered [ 624.117794] LNetError: 28695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 625.194790] LNet: Removed LNI 192.168.203.28@tcp [ 625.920743] Key type .llcrypt unregistered [ 625.923617] Key type ._llcrypt unregistered [ 632.898413] Key type ._llcrypt registered [ 632.901989] Key type .llcrypt registered [ 634.279349] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 01:01:12 (1787115672) [ 634.795947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 634.820203] alg: No test for adler32 (adler32-zlib) [ 635.800540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.868629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 639.210974] LNet: Message forwarding will be enabled in 2s [ 642.271125] LNet: Message forwarding enabled [ 642.820785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 642.840212] LNet: Message forwarding will stop in 2s to 6s [ 647.350685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 647.908406] LNet: Message forwarding disabled [ 650.165680] LNet: Message forwarding will be enabled in 2s [ 654.050730] LNet: Message forwarding enabled [ 657.078938] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 01:01:35 (1787115695) [ 657.205946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 657.233174] LNet: 30098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 658.290235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 658.377767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 665.013526] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 01:01:43 (1787115703) [ 665.381439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 665.408564] LNet: 30684:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 665.432460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.509166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 665.553363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 665.595407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 665.649982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 665.699668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 665.749643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 665.789707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 665.865441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 665.934785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 665.988053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 666.044676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 666.111827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 666.159565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 666.223230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 666.286839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 666.334836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 666.364618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 666.404586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 666.453463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 666.481362] LNetError: 30727:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 673.212791] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 01:01:51 (1787115711) [ 673.311456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 673.337603] LNet: 31290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 673.366845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 673.479752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 679.573628] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 01:01:58 (1787115718) [ 681.428067] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg328-server [ 683.124498] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 01:02:01 (1787115721) [ 683.386826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 683.407832] LNet: 32229:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 683.428494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 683.497994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 683.520841] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 684.305934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[29-33]@tcp [ 684.339704] LNet: 32362:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 684.385358] LNetError: 32233:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.29@tcp is being used as a gateway but routing feature is not turned on [ 684.410361] LNetError: 32233:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.29@tcp has gone from up to down [ 685.415118] LNetError: 32233:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.29@tcp is being used as a gateway but routing feature is not turned on [ 685.430376] LNetError: 32233:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 686.462983] LNetError: 32233:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.33@tcp is being used as a gateway but routing feature is not turned on [ 686.471655] LNetError: 32233:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 688.486981] LNetError: 32233:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.29@tcp is being used as a gateway but routing feature is not turned on [ 688.514244] LNetError: 32233:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 690.196901] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 01:02:09 (1787115729) [ 692.212248] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg328-server [ 692.578811] LNetError: 32233:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.31@tcp is being used as a gateway but routing feature is not turned on [ 692.588029] LNetError: 32233:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 693.534610] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 01:02:12 (1787115732) [ 693.835778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 693.864287] LNet: 33297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 693.872536] LNetError: 33297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 693.891369] LNet: Removed LNI 192.168.203.28@tcp [ 693.911883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 693.967879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 694.000328] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 694.655230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[29-34/2]@tcp [ 694.678267] LNet: 33414:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 694.686671] LNet: 33414:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 694.707878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[29-34/2]@tcp [ 700.549147] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 01:02:19 (1787115739) [ 700.740512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.753890] LNetError: 34007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 700.773220] LNet: Removed LNI 192.168.203.28@tcp [ 700.791494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.868950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 700.915536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.940519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 701.010916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 701.070737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 701.105500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 701.168963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 701.260642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 701.314260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 701.372620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 701.445532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 701.480986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 701.559576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 701.634952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 701.692791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 701.776985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 701.846299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 701.877339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 701.931627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 701.995186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 702.015882] LNet: 34101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 702.020672] LNet: 34101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 702.037922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 702.085860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 702.150612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 702.188994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 702.236728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 702.307122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 702.359304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 702.444461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 702.510818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 702.552240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 702.593983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 702.648656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 702.700798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 702.774832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 708.856904] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 01:02:27 (1787115747) [ 710.826889] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg328-server [ 712.400844] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 01:02:31 (1787115751) [ 714.468346] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg328-server [ 716.184757] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 01:02:35 (1787115755) [ 717.136057] Key type .llcrypt unregistered [ 717.141404] Key type ._llcrypt unregistered [ 723.418236] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 723.641209] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 723.646166] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 723.651806] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 726.199477] Key type ._llcrypt registered [ 726.204894] Key type .llcrypt registered [ 726.345592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 726.359617] alg: No test for adler32 (adler32-zlib) [ 727.269554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 727.321154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 727.355289] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 727.930038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 727.958124] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 729.888311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 729.904678] LNet: Removed LNI 10.1.2.3@tcp [ 730.038421] LNet: 36131:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 730.045270] LNetError: 36131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 730.064069] LNet: Removed LNI 192.168.203.28@tcp [ 730.597434] Key type .llcrypt unregistered [ 730.600972] Key type ._llcrypt unregistered [ 741.746708] Key type ._llcrypt registered [ 741.749978] Key type .llcrypt registered [ 743.036337] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 01:03:01 (1787115781) [ 743.643394] Key type .llcrypt unregistered [ 743.652716] Key type ._llcrypt unregistered [ 750.284624] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 750.446960] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 750.451555] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 750.456933] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 753.045263] Key type ._llcrypt registered [ 753.051795] Key type .llcrypt registered [ 753.259075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 753.275488] alg: No test for adler32 (adler32-zlib) [ 754.208881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 754.294922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 754.320817] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 755.124461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 755.168846] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 757.985080] LNet: 37623:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 757.995944] LNetError: 37623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 758.004874] LNet: Removed LNI 10.1.2.3@tcp [ 758.021483] LNet: Removed LNI 192.168.203.28@tcp [ 758.819540] Key type .llcrypt unregistered [ 758.821408] Key type ._llcrypt unregistered [ 770.056865] Key type ._llcrypt registered [ 770.060589] Key type .llcrypt registered [ 771.042530] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 01:03:30 (1787115810) [ 771.587623] Key type .llcrypt unregistered [ 771.592340] Key type ._llcrypt unregistered [ 776.977391] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 777.056430] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 777.063689] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 777.074089] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 779.588389] Key type ._llcrypt registered [ 779.590926] Key type .llcrypt registered [ 779.722306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 779.742437] alg: No test for adler32 (adler32-zlib) [ 780.645315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 780.719827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 780.752868] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 782.346939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 782.366376] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 784.097194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 784.125595] LNet: Removed LNI 10.1.2.3@tcp [ 784.148603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 784.188702] LNet: Removed LNI 10.1.2.31@tcp [ 784.204638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 784.232642] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 784.766431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 784.788267] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 785.661421] LNet: 39540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 785.666541] LNetError: 39540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 785.671546] LNet: Removed LNI 10.1.2.31@tcp [ 785.679414] LNet: Removed LNI 10.1.2.3@tcp [ 786.235339] Key type .llcrypt unregistered [ 786.238293] Key type ._llcrypt unregistered [ 797.204604] Key type ._llcrypt registered [ 797.208377] Key type .llcrypt registered [ 798.456780] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 01:03:57 (1787115837) [ 798.884356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 798.894782] alg: No test for adler32 (adler32-zlib) [ 799.792815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.886962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 799.929368] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 800.716535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 800.758458] LNet: 40492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 800.774069] LNetError: 40492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 800.801055] LNet: Removed LNI 192.168.203.28@tcp [ 800.830065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 800.890809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.28@tcp [ 800.913203] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 801.676621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.28@tcp [ 801.728200] LNet: Removed LNI 192.168.203.28@tcp [ 801.805348] LNet: 40627:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 802.325491] Key type .llcrypt unregistered [ 802.326898] Key type ._llcrypt unregistered [ 814.014221] Key type ._llcrypt registered [ 814.015567] Key type .llcrypt registered [ 815.336190] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 01:04:13 (1787115853) [ 816.887073] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg328-server [ 818.344074] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 01:04:17 (1787115857) [ 818.889282] Key type .llcrypt unregistered [ 818.892837] Key type ._llcrypt unregistered [ 823.759840] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 823.893961] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 823.901312] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 823.907939] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 826.332304] Key type ._llcrypt registered [ 826.334118] Key type .llcrypt registered [ 826.503817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 826.513766] alg: No test for adler32 (adler32-zlib) [ 827.383152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 827.450666] LNet: Added LNI 192.168.203.28@tcp1 [8/256/0/180] [ 827.994073] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 829.624642] LNet: 42462:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 829.629236] LNetError: 42462:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 829.634883] LNet: Removed LNI 192.168.203.28@tcp1 [ 829.647788] LNet: Removed LNI 10.1.2.3@tcp2 [ 830.186599] Key type .llcrypt unregistered [ 830.195787] Key type ._llcrypt unregistered [ 840.358622] Key type ._llcrypt registered [ 840.360519] Key type .llcrypt registered [ 841.581802] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 01:04:40 (1787115880) [ 841.946837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 841.978283] alg: No test for adler32 (adler32-zlib) [ 842.812509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 842.857277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 842.887652] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 843.506250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 843.532486] LNet: 43430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 843.540548] LNetError: 43430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 843.560153] LNet: Removed LNI 192.168.203.28@tcp [ 843.585514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 843.659589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 843.679807] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 848.874406] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 01:04:47 (1787115887) [ 848.997912] LNet: 44020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 849.008279] LNetError: 44020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 849.024423] LNet: Removed LNI 192.168.203.28@tcp [ 849.479640] Key type .llcrypt unregistered [ 849.484341] Key type ._llcrypt unregistered [ 855.074453] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 855.418807] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 855.429509] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 855.440049] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 858.226545] Key type ._llcrypt registered [ 858.230853] Key type .llcrypt registered [ 858.462332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 858.475158] alg: No test for adler32 (adler32-zlib) [ 859.443364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 859.544774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 859.585310] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 860.428642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 860.461377] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 863.034339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 863.073391] LNet: 44775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 863.080556] LNetError: 44775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 863.088506] LNet: Removed LNI 10.1.2.3@tcp [ 863.096127] LNet: Removed LNI 192.168.203.28@tcp [ 863.127648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 863.200616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 863.246967] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 863.266559] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 863.904565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 863.931254] LNet: 44802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 863.940467] LNetError: 44802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 863.952458] LNetError: 44802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 863.968540] LNet: Removed LNI 10.1.2.3@tcp [ 863.987720] LNet: Removed LNI 192.168.203.28@tcp [ 864.023476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 864.111994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 864.151536] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 864.155578] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 864.870531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 864.907563] LNet: Added LNI 192.168.203.28@tcp2 [8/256/0/180] [ 866.312728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 866.332591] LNet: 44964:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 866.339988] LNetError: 44964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 866.346387] LNetError: 44964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 866.350570] LNet: Removed LNI 10.1.2.3@tcp [ 866.353733] LNet: Removed LNI 192.168.203.28@tcp [ 866.362517] LNet: Removed LNI 192.168.203.28@tcp2 [ 866.386344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.445881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 866.473169] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 866.480835] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 866.485571] LNet: Added LNI 192.168.203.28@tcp2 [8/256/0/180] [ 874.606504] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 01:05:13 (1787115913) [ 874.723226] LNet: 45555:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 874.728761] LNetError: 45555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 874.738430] LNetError: 45555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 874.757321] LNet: Removed LNI 10.1.2.3@tcp [ 874.759178] LNet: Removed LNI 192.168.203.28@tcp [ 874.780393] LNet: Removed LNI 192.168.203.28@tcp2 [ 875.551579] Key type .llcrypt unregistered [ 875.554293] Key type ._llcrypt unregistered [ 881.371923] Key type ._llcrypt registered [ 881.375109] Key type .llcrypt registered [ 881.647705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 881.679476] alg: No test for adler32 (adler32-zlib) [ 882.579329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.645083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 882.672202] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 883.648755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 883.688302] LNet: 45956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.695392] LNetError: 45956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.710449] LNet: Removed LNI 192.168.203.28@tcp [ 883.732577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.824281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 883.878087] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 884.416394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 884.436493] LNet: 45986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 884.441646] LNetError: 45986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 884.467683] LNet: Removed LNI 192.168.203.28@tcp [ 884.487494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.553913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 884.575261] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 884.979639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 885.017991] LNet: Removed LNI 192.168.203.28@tcp [ 885.036404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 885.090664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 885.115689] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 885.618090] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 885.724582] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 885.732989] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 885.739060] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 888.117681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 888.139620] LNet: 46096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 888.145937] LNet: 46096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 888.152628] LNetError: 46096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 888.155741] LNetError: 46096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 888.187651] LNet: Removed LNI 192.168.203.28@tcp [ 888.208863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 888.275256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 888.298345] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 894.270867] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 01:05:33 (1787115933) [ 894.507250] LNet: 46694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 894.523616] LNetError: 46694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 894.548988] LNet: Removed LNI 192.168.203.28@tcp [ 895.198893] Key type .llcrypt unregistered [ 895.205570] Key type ._llcrypt unregistered [ 901.434364] Key type ._llcrypt registered [ 901.436850] Key type .llcrypt registered [ 901.670422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 901.689861] alg: No test for adler32 (adler32-zlib) [ 902.600448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 902.670827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 902.698534] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 903.417471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 903.450300] LNet: 47092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 903.456754] LNetError: 47092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 903.470581] LNet: Removed LNI 192.168.203.28@tcp [ 903.487475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 903.553948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 903.576364] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 909.209529] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 01:05:48 (1787115948) [ 909.318158] LNet: 47680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.331724] LNetError: 47680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 909.341294] LNet: Removed LNI 192.168.203.28@tcp [ 909.860497] Key type .llcrypt unregistered [ 909.863066] Key type ._llcrypt unregistered [ 915.475796] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 915.646559] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 915.656476] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 915.666760] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 918.293897] Key type ._llcrypt registered [ 918.295664] Key type .llcrypt registered [ 918.481246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 918.493345] alg: No test for adler32 (adler32-zlib) [ 919.380226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 919.428295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 919.448142] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 920.397957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 920.414297] LNet: 48151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.418707] LNetError: 48151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 920.430236] LNet: Removed LNI 192.168.203.28@tcp [ 920.446489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 920.509492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 920.533793] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 926.888784] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 01:06:05 (1787115965) [ 926.971818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 927.000851] LNet: 48744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 927.006745] LNetError: 48744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 927.036392] LNet: Removed LNI 192.168.203.28@tcp [ 927.072453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 927.201935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 932.965510] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 01:06:11 (1787115971) [ 933.151614] LNet: 49329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 933.756746] Key type .llcrypt unregistered [ 933.760504] Key type ._llcrypt unregistered [ 939.486571] Key type ._llcrypt registered [ 939.489615] Key type .llcrypt registered [ 939.669411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 939.696634] alg: No test for adler32 (adler32-zlib) [ 940.645096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 940.707977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 940.729969] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 941.556224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 941.575214] LNet: 49730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 941.581899] LNetError: 49730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 941.596730] LNet: Removed LNI 192.168.203.28@tcp [ 941.629216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 941.702392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 941.726963] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 948.117774] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 01:06:27 (1787115987) [ 948.444930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 948.484535] LNet: 50323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.496535] LNetError: 50323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 948.544808] LNet: Removed LNI 192.168.203.28@tcp [ 948.569489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 948.671549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 954.230670] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 01:06:33 (1787115993) [ 955.780479] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 957.303686] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 01:06:36 (1787115996) [ 958.523644] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 960.048155] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 01:06:39 (1787115999) [ 961.359919] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 962.779831] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 01:06:41 (1787116001) [ 963.896432] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 965.436855] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 01:06:44 (1787116004) [ 966.792712] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 968.273165] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 01:06:47 (1787116007) [ 968.364852] LNet: 52366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 968.977779] Key type .llcrypt unregistered [ 968.980994] Key type ._llcrypt unregistered [ 974.132647] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 974.324750] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 974.330252] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 974.335313] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 976.816376] Key type ._llcrypt registered [ 976.818573] Key type .llcrypt registered [ 977.021793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 977.041693] alg: No test for adler32 (adler32-zlib) [ 977.947259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 978.029100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 978.045818] LNet: Added LNI 192.168.203.28@tcp [8/512/0/180] [ 978.049735] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 978.052110] LNet: 52695:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 978.150961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 978.169889] LNet: 52714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 978.174595] LNetError: 52714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 978.179114] LNet: Removed LNI 10.1.2.3@tcp [ 978.181094] LNetError: 52714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 978.193268] LNet: Removed LNI 192.168.203.28@tcp [ 978.221386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 978.300796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 978.325071] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 978.396327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 978.422268] LNet: 52738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 978.435818] LNetError: 52738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 978.458689] LNet: Removed LNI 192.168.203.28@tcp [ 978.485897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 978.530792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 978.549269] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 978.552931] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 978.555408] LNet: 52746:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 978.635962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 978.654914] LNet: 52764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 978.662803] LNetError: 52764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 978.667687] LNet: Removed LNI 10.1.2.3@tcp [ 978.671992] LNetError: 52764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 978.692931] LNet: Removed LNI 192.168.203.28@tcp [ 978.728048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 978.786840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 978.826668] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 978.832076] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 978.838138] LNet: 52772:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 979.063519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 979.101564] LNet: 52798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 979.108187] LNetError: 52798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 979.119616] LNet: Removed LNI 10.1.2.3@tcp [ 979.121401] LNetError: 52798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 979.142688] LNet: Removed LNI 192.168.203.28@tcp [ 979.187438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 979.278877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 979.298068] LNet: Added LNI 192.168.203.28@tcp [8/256/0/182] [ 979.305170] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 979.423503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 979.455100] LNet: 52826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 979.459163] LNetError: 52826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 979.467355] LNet: Removed LNI 10.1.2.3@tcp [ 979.470499] LNetError: 52826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 979.492700] LNet: Removed LNI 192.168.203.28@tcp [ 979.513628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 979.581336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 979.605394] LNet: Added LNI 192.168.203.28@tcp [8/256/0/182] [ 979.613625] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 979.617833] LNet: 52834:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 979.698805] LNet: Removed LNI 10.1.2.3@tcp [ 979.710717] LNet: Removed LNI 192.168.203.28@tcp [ 980.364226] Key type .llcrypt unregistered [ 980.366047] Key type ._llcrypt unregistered [ 990.331935] Key type ._llcrypt registered [ 990.335190] Key type .llcrypt registered [ 991.818952] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 01:07:10 (1787116030) [ 992.343275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 992.360540] alg: No test for adler32 (adler32-zlib) [ 993.315137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 993.384684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 993.425251] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 994.214633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.28 [ 994.238662] LNet: 53810:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 994.244562] LNet: Removed LNI 192.168.203.28@tcp [ 999.795472] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 01:07:18 (1787116038) [ 999.916901] LNet: 54375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 999.936770] LNetError: 54375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 999.961460] LNet: Removed LNI 192.168.203.28@tcp [ 1000.671389] Key type .llcrypt unregistered [ 1000.673752] Key type ._llcrypt unregistered [ 1006.096640] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1006.275888] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1008.956188] Key type ._llcrypt registered [ 1008.958758] Key type .llcrypt registered [ 1009.116442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1009.155780] alg: No test for adler32 (adler32-zlib) [ 1010.437189] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1016.245542] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 01:07:35 (1787116055) [ 1016.559740] LNet: 55295:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1016.571673] LNetError: 55295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1016.595994] LNet: Removed LNI 10.1.2.3@tcp [ 1017.148337] Key type .llcrypt unregistered [ 1017.150943] Key type ._llcrypt unregistered [ 1022.981463] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1023.185691] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1023.189936] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1023.198809] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1025.815140] Key type ._llcrypt registered [ 1025.818500] Key type .llcrypt registered [ 1025.967933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1025.980219] alg: No test for adler32 (adler32-zlib) [ 1026.893435] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1032.165416] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 01:07:51 (1787116071) [ 1033.406826] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1034.862627] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 01:07:53 (1787116073) [ 1035.105957] LNet: 56478:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1035.117216] LNetError: 56478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1035.140057] LNet: Removed LNI 10.1.2.3@tcp [ 1035.845606] Key type .llcrypt unregistered [ 1035.847566] Key type ._llcrypt unregistered [ 1041.418527] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1041.660961] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1041.667857] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1041.671803] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1044.283461] Key type ._llcrypt registered [ 1044.287948] Key type .llcrypt registered [ 1044.459139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1044.486618] alg: No test for adler32 (adler32-zlib) [ 1045.414130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1045.487463] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1052.847654] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 01:08:11 (1787116091) [ 1053.323499] LNet: 57707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1053.329340] LNetError: 57707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1053.350416] LNet: Removed LNI 10.1.2.3@tcp [ 1054.032159] Key type .llcrypt unregistered [ 1054.038109] Key type ._llcrypt unregistered [ 1059.915160] Key type ._llcrypt registered [ 1059.920975] Key type .llcrypt registered [ 1060.205863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1060.232414] alg: No test for adler32 (adler32-zlib) [ 1061.175313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1061.205885] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1067.693422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.128@tcp [ 1068.246854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e local_interrupt -r 1 [ 1068.283728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e local_interrupt -r 1 [ 1068.329756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.128@tcp [ 1068.355405] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1068.383796] LNet: 1 local NIs in recovery (showing 1): 192.168.203.28@tcp [ 1068.581532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1068.607791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1068.634814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1068.668941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1068.701319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1068.760195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1068.822558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1068.876794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1068.919168] LNet: 58145:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1068.943371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1068.988907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1069.037287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1069.074097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1069.098652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1069.150616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1069.200714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1069.920361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e local_dropped -r 1 [ 1069.988740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e local_dropped -r 1 [ 1070.069964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.128@tcp [ 1070.101262] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1070.349327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1070.386288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1070.417966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1070.451588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1070.505813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1070.541534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1070.580950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1070.615736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1070.639433] LNet: 58254:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1070.656604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1070.685573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1070.713885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1070.748639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1070.780117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1070.813287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1070.843412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1071.335627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e local_aborted -r 1 [ 1071.385099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e local_aborted -r 1 [ 1071.440941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.128@tcp [ 1071.476854] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1071.706718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1071.753786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1071.815622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1071.849952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1071.889372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1071.934431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1071.971520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1072.005924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1072.040585] LNet: 58363:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1072.064399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1072.089635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1072.137228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1072.180458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1072.215610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1072.252719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1072.295946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1072.643538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e local_no_route -r 1 [ 1072.667908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e local_no_route -r 1 [ 1072.692556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.128@tcp [ 1072.886355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1072.919563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1072.947110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1072.975168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1073.006225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1073.031751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1073.057515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1073.082587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1073.110752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1073.139675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1073.169904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1073.202781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1073.235342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1073.282774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1073.325272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1073.830068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e local_timeout -r 1 [ 1073.881806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e local_timeout -r 1 [ 1073.918794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.128@tcp [ 1073.950202] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1073.956338] LNet: Skipped 1 previous similar message [ 1074.179434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1074.212140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1074.238683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1074.267548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1074.304658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1074.344101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1074.371322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1074.400784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1074.420205] LNet: 58582:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1074.426659] LNet: 58582:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1074.444866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1074.473621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1074.507356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1074.576843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1074.619395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1074.672911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1074.701287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1075.150534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e local_error -r 1 [ 1075.193970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e local_error -r 1 [ 1075.225723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.128@tcp [ 1075.428403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1075.457492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1075.485464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1075.513487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1075.538196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1075.565255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1075.590647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1075.616552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1075.646882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1075.680171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1075.726511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1075.762704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1075.788477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1075.820721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1075.884047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1076.118211] LNet: 58732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1076.125324] LNetError: 58732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1076.139961] LNet: Removed LNI 192.168.203.28@tcp [ 1076.419682] Key type .llcrypt unregistered [ 1076.426497] Key type ._llcrypt unregistered [ 1091.237654] Key type ._llcrypt registered [ 1091.239887] Key type .llcrypt registered [ 1092.569697] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 01:08:51 (1787116131) [ 1093.446305] Key type .llcrypt unregistered [ 1093.451811] Key type ._llcrypt unregistered [ 1099.701312] Key type ._llcrypt registered [ 1099.703910] Key type .llcrypt registered [ 1099.878198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1099.891960] alg: No test for adler32 (adler32-zlib) [ 1100.791536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1100.823428] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1107.763787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.128@tcp [ 1118.583946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1118.638317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1118.654187] LNet: Added LNI 192.168.203.28@tcp1 [8/256/0/180] [ 1119.983750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e local_interrupt -r 1 [ 1120.015200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp1 -m GET -e local_interrupt -r 1 [ 1120.044878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e local_interrupt -r 1 [ 1120.080654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp1 -m GET -e local_interrupt -r 1 [ 1120.112376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp -m GET -e local_interrupt -r 1 [ 1120.162691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp1 -m GET -e local_interrupt -r 1 [ 1120.190091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp -m GET -e local_interrupt -r 1 [ 1120.221813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp1 -m GET -e local_interrupt -r 1 [ 1120.289859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.128@tcp [ 1120.333924] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1120.347024] LNet: 1 local NIs in recovery (showing 1): 192.168.203.28@tcp [ 1120.519052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1120.538954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1120.556255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1120.577725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1120.611822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1120.644358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1120.669946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1120.692475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1120.710465] LNet: 60202:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1120.722984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1120.753299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1120.784154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1120.812146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1120.835170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1120.860740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1120.884693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1121.494526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e local_dropped -r 1 [ 1121.549398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp1 -m GET -e local_dropped -r 1 [ 1121.594408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e local_dropped -r 1 [ 1121.626454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp1 -m GET -e local_dropped -r 1 [ 1121.656785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp -m GET -e local_dropped -r 1 [ 1121.686096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp1 -m GET -e local_dropped -r 1 [ 1121.724750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp -m GET -e local_dropped -r 1 [ 1121.780780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp1 -m GET -e local_dropped -r 1 [ 1121.845162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.128@tcp [ 1121.889510] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1121.900161] LNet: Skipped 2 previous similar messages [ 1122.205917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1122.249341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1122.300689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1122.326933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1122.362610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1122.383092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1122.411339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1122.436624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1122.447239] LNet: 60323:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1122.460255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1122.480332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1122.504538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1122.561289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1122.611481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1122.638400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1122.665471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1123.283895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e local_aborted -r 1 [ 1123.348758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp1 -m GET -e local_aborted -r 1 [ 1123.387531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e local_aborted -r 1 [ 1123.435138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp1 -m GET -e local_aborted -r 1 [ 1123.465278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp -m GET -e local_aborted -r 1 [ 1123.494636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp1 -m GET -e local_aborted -r 1 [ 1123.518308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp -m GET -e local_aborted -r 1 [ 1123.548210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp1 -m GET -e local_aborted -r 1 [ 1123.583825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.128@tcp [ 1123.600858] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1123.606227] LNet: Skipped 2 previous similar messages [ 1123.810219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1123.860188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1123.891904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1123.926674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1123.957487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1123.982354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1124.003745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1124.026677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1124.045749] LNet: 60444:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1124.064683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1124.096188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1124.114722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1124.151658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1124.212567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1124.272452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1124.303711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1124.883774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e local_no_route -r 1 [ 1124.980968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp1 -m GET -e local_no_route -r 1 [ 1125.056185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e local_no_route -r 1 [ 1125.119576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp1 -m GET -e local_no_route -r 1 [ 1125.170727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp -m GET -e local_no_route -r 1 [ 1125.221994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp1 -m GET -e local_no_route -r 1 [ 1125.280570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp -m GET -e local_no_route -r 1 [ 1125.323491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp1 -m GET -e local_no_route -r 1 [ 1125.369886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.128@tcp [ 1125.677819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1125.746423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1125.822184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1125.874881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1125.915834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1125.944949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1125.990535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1126.039093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1126.065679] LNet: 60566:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1126.115098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1126.189771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1126.222722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1126.255182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1126.292623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1126.355864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1126.393992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1127.047089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e local_timeout -r 1 [ 1127.082697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp1 -m GET -e local_timeout -r 1 [ 1127.130117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e local_timeout -r 1 [ 1127.174429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp1 -m GET -e local_timeout -r 1 [ 1127.222862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp -m GET -e local_timeout -r 1 [ 1127.265603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp1 -m GET -e local_timeout -r 1 [ 1127.305587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp -m GET -e local_timeout -r 1 [ 1127.341461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp1 -m GET -e local_timeout -r 1 [ 1127.374560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.128@tcp [ 1127.393573] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1127.400083] LNet: Skipped 5 previous similar messages [ 1127.599884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1127.619844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1127.661896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1127.683329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1127.711666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1127.736619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1127.760228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1127.780496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1127.803215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1127.838350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1127.857735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1127.876276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1127.895573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1127.915849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1127.939763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1128.410811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e local_error -r 1 [ 1128.462225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp1 -m GET -e local_error -r 1 [ 1128.499645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e local_error -r 1 [ 1128.549877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp1 -m GET -e local_error -r 1 [ 1128.603393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp -m GET -e local_error -r 1 [ 1128.670490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp1 -m GET -e local_error -r 1 [ 1128.718488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp -m GET -e local_error -r 1 [ 1128.765501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp1 -m GET -e local_error -r 1 [ 1128.811898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.128@tcp [ 1128.992889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1129.023394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1129.048898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1129.075787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1129.099711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1129.123816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1129.149545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1129.197574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1129.254865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1129.297737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1129.325917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1129.373635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1129.415603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1129.451284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1129.490907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1135.362878] LNet: 60885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1135.375137] LNetError: 60885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1135.383754] LNet: Removed LNI 192.168.203.28@tcp [ 1136.426741] LNet: Removed LNI 192.168.203.28@tcp1 [ 1137.066448] Key type .llcrypt unregistered [ 1137.075041] Key type ._llcrypt unregistered [ 1155.055181] Key type ._llcrypt registered [ 1155.060954] Key type .llcrypt registered [ 1156.622260] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 01:09:54 (1787116194) [ 1157.275436] Key type .llcrypt unregistered [ 1157.284083] Key type ._llcrypt unregistered [ 1162.333972] Key type ._llcrypt registered [ 1162.338162] Key type .llcrypt registered [ 1162.474169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1162.483927] alg: No test for adler32 (adler32-zlib) [ 1163.417361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1163.457620] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1171.900322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.128@tcp [ 1172.780549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e remote_dropped -r 1 [ 1172.810683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e remote_dropped -r 1 [ 1172.848834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.128@tcp [ 1172.885151] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1173.094446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1173.130411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1173.173610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1173.223712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1173.281757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1173.334956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1173.388776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1173.444644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1173.478637] LNet: 62134:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1173.527737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1173.583086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1173.633367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1173.692855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1173.762710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1173.828964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1173.898908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1174.561896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e remote_error -r 1 [ 1174.592739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e remote_error -r 1 [ 1174.624584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.128@tcp [ 1174.644712] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1174.842292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1174.870937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1174.896741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1174.935670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1174.974572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1175.007409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1175.042161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1175.099650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1175.138750] LNet: 62244:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1175.158892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1175.193975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1175.237405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1175.298135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1175.339703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1175.394543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1175.432912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1176.010155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e remote_timeout -r 1 [ 1176.037993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e remote_timeout -r 1 [ 1176.084855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.128@tcp [ 1176.101383] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1176.363804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1176.407195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1176.434810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1176.519997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1176.555757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1176.589174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1176.616584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1176.640668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1176.655326] LNet: 62353:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1176.671708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1176.693583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1176.719396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1176.740195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1176.772765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1176.797834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1176.828680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1177.039326] LNet: 62394:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1177.045706] LNetError: 62394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1177.061939] LNet: Removed LNI 192.168.203.28@tcp [ 1177.398703] Key type .llcrypt unregistered [ 1177.403261] Key type ._llcrypt unregistered [ 1194.565678] Key type ._llcrypt registered [ 1194.567770] Key type .llcrypt registered [ 1196.351889] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 01:10:34 (1787116234) [ 1197.582418] Key type .llcrypt unregistered [ 1197.585416] Key type ._llcrypt unregistered [ 1204.926806] Key type ._llcrypt registered [ 1204.930785] Key type .llcrypt registered [ 1205.102280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1205.115488] alg: No test for adler32 (adler32-zlib) [ 1206.117449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1206.172649] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1213.340926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.128@tcp [ 1222.887983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1222.960301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1223.010835] LNet: Added LNI 192.168.203.28@tcp1 [8/256/0/180] [ 1224.703670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e remote_dropped -r 1 [ 1224.735785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp1 -m GET -e remote_dropped -r 1 [ 1224.774562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e remote_dropped -r 1 [ 1224.811716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp1 -m GET -e remote_dropped -r 1 [ 1224.848558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp -m GET -e remote_dropped -r 1 [ 1224.890628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp1 -m GET -e remote_dropped -r 1 [ 1224.941025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp -m GET -e remote_dropped -r 1 [ 1224.985904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp1 -m GET -e remote_dropped -r 1 [ 1225.040403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.128@tcp [ 1225.060919] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1225.066718] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.128@tcp [ 1225.332361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1225.361629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1225.385862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1225.417458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1225.438433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1225.477242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1225.518384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1225.551689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1225.576198] LNet: 63862:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1225.595574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1225.628806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1225.663116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1225.718547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1225.758357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1225.791970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1225.823955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1225.862437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1226.312875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e remote_error -r 1 [ 1226.351630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp1 -m GET -e remote_error -r 1 [ 1226.386553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e remote_error -r 1 [ 1226.417330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp1 -m GET -e remote_error -r 1 [ 1226.449896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp -m GET -e remote_error -r 1 [ 1226.476417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp1 -m GET -e remote_error -r 1 [ 1226.513526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp -m GET -e remote_error -r 1 [ 1226.553255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp1 -m GET -e remote_error -r 1 [ 1226.586151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.128@tcp [ 1226.606680] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1226.616463] LNet: Skipped 2 previous similar messages [ 1226.794867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1226.821404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1226.854462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1226.883347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1226.923436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1226.958835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1226.990365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1227.018137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1227.036751] LNet: 63986:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1227.052662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1227.086903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1227.130736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1227.165881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1227.205159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1227.245151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1227.290838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1227.327624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1227.781532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e remote_timeout -r 1 [ 1227.826720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp1 -m GET -e remote_timeout -r 1 [ 1227.863505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e remote_timeout -r 1 [ 1227.922067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp1 -m GET -e remote_timeout -r 1 [ 1227.974730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp -m GET -e remote_timeout -r 1 [ 1228.042373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp1 -m GET -e remote_timeout -r 1 [ 1228.105337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp -m GET -e remote_timeout -r 1 [ 1228.156319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp1 -m GET -e remote_timeout -r 1 [ 1228.194384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.128@tcp [ 1228.213054] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1228.219411] LNet: Skipped 1 previous similar message [ 1228.434185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1228.456620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1228.502644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1228.549599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1228.592700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1228.635368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1228.674133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1228.699742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1228.718744] LNet: 64109:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1228.735877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1228.787983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1228.838227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1228.872191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1228.894385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1228.920408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1228.952380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1228.978716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1233.829369] LNet: 64187:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1233.837488] LNetError: 64187:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1233.843843] LNet: Removed LNI 192.168.203.28@tcp [ 1233.856689] LNet: Removed LNI 192.168.203.28@tcp1 [ 1234.428866] Key type .llcrypt unregistered [ 1234.432229] Key type ._llcrypt unregistered [ 1249.337987] Key type ._llcrypt registered [ 1249.340496] Key type .llcrypt registered [ 1250.439062] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 01:11:29 (1787116289) [ 1251.730984] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1253.292984] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 01:11:32 (1787116292) [ 1253.972286] Key type .llcrypt unregistered [ 1253.975574] Key type ._llcrypt unregistered [ 1260.068637] Key type ._llcrypt registered [ 1260.073098] Key type .llcrypt registered [ 1260.311167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1260.337905] alg: No test for adler32 (adler32-zlib) [ 1261.334674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1261.377469] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1269.095636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.128@tcp [ 1269.743763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e network_timeout -r 1 [ 1269.784636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e network_timeout -r 1 [ 1269.816635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.128@tcp [ 1269.833253] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1270.094460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1270.139292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1270.171899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1270.207423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1270.248477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1270.278226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1270.324864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1270.356043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1270.376538] LNet: 65730:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1270.394340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1270.429421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1270.467509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1270.499223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1270.528426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1270.562878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1270.898743] LNet: 65769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1270.908956] LNetError: 65769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1270.937456] LNet: Removed LNI 192.168.203.28@tcp [ 1271.291630] Key type .llcrypt unregistered [ 1271.296408] Key type ._llcrypt unregistered [ 1288.188454] Key type ._llcrypt registered [ 1288.191738] Key type .llcrypt registered [ 1288.426734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1288.460685] alg: No test for adler32 (adler32-zlib) [ 1289.396860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1289.414285] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1295.874179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.128@tcp [ 1304.260237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1304.313066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1304.342681] LNet: Added LNI 192.168.203.28@tcp1 [8/256/0/180] [ 1305.741964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e network_timeout -r 1 [ 1305.776310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp1 -m GET -e network_timeout -r 1 [ 1305.829250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e network_timeout -r 1 [ 1305.879446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp1 -m GET -e network_timeout -r 1 [ 1305.930808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp -m GET -e network_timeout -r 1 [ 1305.982605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp1 -m GET -e network_timeout -r 1 [ 1306.014855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp -m GET -e network_timeout -r 1 [ 1306.057221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp1 -m GET -e network_timeout -r 1 [ 1306.120407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.128@tcp [ 1306.147854] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1306.449496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1306.488659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1306.543470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1306.603066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1306.641755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1306.677698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1306.719430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1306.756144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1306.773421] LNet: 66649:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1306.801953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1306.859777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1306.862161] LNet: 1 local NIs in recovery (showing 1): 192.168.203.28@tcp [ 1306.881293] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.128@tcp [ 1306.897303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1306.945029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1307.007914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1307.046680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1311.258896] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1311.629892] LNet: 66723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1311.652780] LNetError: 66723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1311.666274] LNet: Removed LNI 192.168.203.28@tcp [ 1311.686744] LNet: Removed LNI 192.168.203.28@tcp1 [ 1312.107193] Key type .llcrypt unregistered [ 1312.112266] Key type ._llcrypt unregistered [ 1326.916505] Key type ._llcrypt registered [ 1326.921246] Key type .llcrypt registered [ 1327.960333] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 01:12:46 (1787116366) [ 1328.652450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1328.678947] alg: No test for adler32 (adler32-zlib) [ 1329.563780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1329.604536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1329.636667] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1330.336237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1330.370868] LNet: Added LNI 192.168.203.28@tcp1 [8/256/0/180] [ 1331.400620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 1331.500675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1331.577362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1331.618453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1331.657595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1331.698338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1331.735936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.28@tcp [ 1331.761798] LNetError: 67825:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.28@tcp to recovery [ 1332.641705] LNet: 1 local NIs in recovery (showing 1): 192.168.203.28@tcp [ 1332.642684] LNet: There was an unexpected network error while writing to 192.168.203.28: rc = -22 [ 1334.688861] LNet: There was an unexpected network error while writing to 192.168.203.28: rc = -22 [ 1335.319374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1338.783829] LNet: There was an unexpected network error while writing to 192.168.203.28: rc = -22 [ 1339.904416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1339.929827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1340.015404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1340.084709] LNet: 67943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1340.095099] LNetError: 67943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1340.106658] LNet: Removed LNI 192.168.203.28@tcp [ 1340.128316] LNet: Removed LNI 192.168.203.28@tcp1 [ 1340.146363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1340.220470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1340.240414] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1341.015991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1341.027981] LNet: Added LNI 192.168.203.28@tcp1 [8/256/0/180] [ 1341.827464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 1341.858457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1341.905752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1341.938863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1341.969868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1341.997542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1342.026884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1342.054117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.28@tcp [ 1342.091487] LNetError: 68215:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.28@tcp to recovery [ 1342.248230] LNet: There was an unexpected network error while writing to 192.168.203.28: rc = -22 [ 1344.553377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1348.383496] LNet: There was an unexpected network error while writing to 192.168.203.28: rc = -22 [ 1348.401505] LNet: Skipped 1 previous similar message [ 1353.618405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1353.679039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1353.712048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1360.292579] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 01:13:18 (1787116398) [ 1360.469342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1360.524511] LNet: 68929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1360.544217] LNetError: 68929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1360.558710] LNetError: 68929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1360.572225] LNet: Removed LNI 192.168.203.28@tcp [ 1360.584391] LNet: Removed LNI 192.168.203.28@tcp1 [ 1360.665910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1360.749226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1360.770496] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1361.475345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1361.513560] LNet: Added LNI 192.168.203.28@tcp1 [8/256/0/180] [ 1362.542342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 1362.634659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1362.664596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1362.733692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1362.793901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1362.824419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1362.854807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.28@tcp --health 0 [ 1363.009214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1363.807331] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.28@tcp [ 1363.809052] LNet: There was an unexpected network error while writing to 192.168.203.28: rc = -22 [ 1363.842294] LNet: Skipped 1 previous similar message [ 1377.587940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1377.844181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1377.900160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1377.936085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.28@tcp --health 500 [ 1378.067831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1383.737661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1383.794607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1383.830841] LNet: 69409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1383.840474] LNetError: 69409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1383.849297] LNetError: 69409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1383.857045] LNet: Removed LNI 192.168.203.28@tcp [ 1383.868485] LNet: Removed LNI 192.168.203.28@tcp1 [ 1383.888427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1383.952822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1383.988524] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1384.942430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1384.963257] LNet: Added LNI 192.168.203.28@tcp1 [8/256/0/180] [ 1385.860539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 1385.981407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1386.097662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1386.153724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1386.222645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1386.270454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1386.322915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1386.381534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.28@tcp --health 0 [ 1387.999387] LNet: There was an unexpected network error while writing to 192.168.203.28: rc = -22 [ 1388.010816] LNet: Skipped 3 previous similar messages [ 1388.913949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1398.040894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1398.072401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1398.102655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1405.056921] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 01:14:03 (1787116443) [ 1405.690548] LNet: 70455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1405.700614] LNetError: 70455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1405.718406] LNetError: 70455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1405.731803] LNet: Removed LNI 192.168.203.28@tcp [ 1405.758510] LNet: Removed LNI 192.168.203.28@tcp1 [ 1406.306897] Key type .llcrypt unregistered [ 1406.309320] Key type ._llcrypt unregistered [ 1411.792491] Key type ._llcrypt registered [ 1411.795408] Key type .llcrypt registered [ 1411.950225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1411.988491] alg: No test for adler32 (adler32-zlib) [ 1412.919356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1412.944713] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1421.059605] Lustre: DEBUG MARKER: Initial discovery [ 1421.122553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.128@tcp [ 1423.258258] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1423.362431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1423.398811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.128@tcp [ 1423.427501] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1423.452413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1423.499311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.128@tcp -r 1 -m GET -l 3 [ 1423.561415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.128@tcp -r 1 -m GET -e local_error [ 1423.596983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.128@tcp -r 1 -m PUT -l 6 [ 1424.236279] LNet: 1 local NIs in recovery (showing 1): 192.168.203.28@tcp [ 1424.981594] Lustre: DEBUG MARKER: Force oleg328-client.virtnet to discover 192.168.203.128@tcp (in background) [ 1425.055253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.128@tcp [ 1426.808779] Lustre: DEBUG MARKER: Force oleg328-server to discover 192.168.203.28@tcp [ 1428.127226] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1429.415810] Lustre: DEBUG MARKER: Wait for 71211 [ 1431.573513] Lustre: DEBUG MARKER: Finished wait on 71211 [ 1431.738483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1431.801192] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1431.815360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1432.220761] LNet: 71670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1432.227874] LNetError: 71670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1432.257777] LNet: Removed LNI 192.168.203.28@tcp [ 1432.806455] Key type .llcrypt unregistered [ 1432.810803] Key type ._llcrypt unregistered [ 1451.288106] Key type ._llcrypt registered [ 1451.298855] Key type .llcrypt registered [ 1452.849666] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 01:14:51 (1787116491) [ 1453.420385] Key type .llcrypt unregistered [ 1453.423302] Key type ._llcrypt unregistered [ 1459.382616] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1459.461636] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1459.464567] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1459.470042] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1461.898632] Key type ._llcrypt registered [ 1461.900566] Key type .llcrypt registered [ 1462.062318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1462.085304] alg: No test for adler32 (adler32-zlib) [ 1462.977239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1463.072633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1463.115898] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1463.898856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1463.918652] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1471.551529] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 01:15:10 (1787116510) [ 1471.737454] LNet: 73740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1471.759169] LNetError: 73740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1471.773121] LNet: Removed LNI 10.1.2.3@tcp [ 1471.791976] LNet: Removed LNI 192.168.203.28@tcp [ 1472.405497] Key type .llcrypt unregistered [ 1472.410336] Key type ._llcrypt unregistered [ 1478.877781] Key type ._llcrypt registered [ 1478.880760] Key type .llcrypt registered [ 1479.016385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1479.030761] alg: No test for adler32 (adler32-zlib) [ 1479.940265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1479.994232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1480.013841] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1480.741371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1480.760969] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1482.005535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.28@tcp [ 1483.104391] LNetError: Unexpected error -22 connecting to 192.168.203.28@tcp at host 192.168.203.28:988 [ 1488.289648] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 01:15:27 (1787116527) [ 1488.487644] LNet: 74874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1488.498602] LNetError: 74874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1488.508221] LNet: Removed LNI 10.1.2.3@tcp [ 1488.526916] LNet: Removed LNI 192.168.203.28@tcp [ 1489.199946] Key type .llcrypt unregistered [ 1489.203338] Key type ._llcrypt unregistered [ 1495.469955] Key type ._llcrypt registered [ 1495.473202] Key type .llcrypt registered [ 1495.697883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1495.721414] alg: No test for adler32 (adler32-zlib) [ 1496.638366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1496.759798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1496.806988] LNet: Added LNI 192.168.203.28@tcp1 [8/256/0/180] [ 1497.555093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1497.592147] LNet: Added LNI 192.168.203.28@tcp2 [8/256/0/180] [ 1498.518308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.28@tcp1 --nid 192.168.203.28@tcp2 [ 1502.562928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.28@tcp1 192.168.203.28@tcp2 [ 1502.582320] LNetError: 75709:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.28@tcp1 to 192.168.203.28@tcp2 - Route table may be misconfigured [ 1502.612697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.28@tcp2 192.168.203.28@tcp1 [ 1508.713349] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 01:15:47 (1787116547) [ 1508.838058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1508.867385] LNet: 76281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1508.878581] LNetError: 76281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1509.919159] LNet: Removed LNI 192.168.203.28@tcp1 [ 1509.924711] LNetError: 76281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1509.957510] LNet: Removed LNI 192.168.203.28@tcp2 [ 1509.995035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1510.051847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1510.101537] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1510.836943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1510.859950] LNet: Added LNI 192.168.203.28@tcp1 [8/256/0/180] [ 1511.797258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 1511.855453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.28@tcp 192.168.203.28@tcp [ 1511.902249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.28@tcp1 192.168.203.28@tcp1 [ 1511.960667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.28@tcp -d 192.168.203.28@tcp -e network_timeout [ 1512.012247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.28@tcp -d 192.168.203.28@tcp1 -e network_timeout [ 1512.069425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp -e network_timeout [ 1512.112766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp1 -e network_timeout [ 1512.193282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 1512.238346] LNet: There was an unexpected network error while writing to 192.168.203.28: rc = -22 [ 1512.282307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1512.380185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1512.469498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1513.119250] LNet: 1 local NIs in recovery (showing 1): 192.168.203.28@tcp [ 1518.165478] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 01:15:57 (1787116557) [ 1518.340773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1518.378912] LNet: 77144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1518.394800] LNetError: 77144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1519.455151] LNet: Removed LNI 192.168.203.28@tcp [ 1520.494826] LNet: Removed LNI 192.168.203.28@tcp1 [ 1520.532501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1520.633544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1520.900450] LNet: 77183:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1521.331887] Key type .llcrypt unregistered [ 1521.335993] Key type ._llcrypt unregistered [ 1532.174227] Key type ._llcrypt registered [ 1532.176433] Key type .llcrypt registered [ 1533.336068] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 01:16:12 (1787116572) [ 1533.870560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1533.903207] alg: No test for adler32 (adler32-zlib) [ 1534.964430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1536.913037] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1538.679758] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 01:16:17 (1787116577) [ 1538.767875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1538.788094] LNet: 78274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1538.822824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1538.877964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1538.899595] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1539.488893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1539.527609] LNet: Added LNI 192.168.203.28@tcp1 [8/256/0/180] [ 1540.403957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 1540.442790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp1 [ 1540.468361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp1 [ 1547.703124] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 01:16:26 (1787116586) [ 1549.765590] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg328-server [ 1552.259872] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 01:16:30 (1787116590) [ 1554.553368] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg328-server [ 1556.713420] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 01:16:35 (1787116595) [ 1558.659282] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg328-server [ 1560.860323] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 01:16:39 (1787116599) [ 1562.793685] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg328-server [ 1564.587054] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 01:16:43 (1787116603) [ 1566.283989] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg328-server [ 1567.831227] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 01:16:46 (1787116606) [ 1569.691221] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg328-server [ 1571.635205] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 01:16:50 (1787116610) [ 1573.368372] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg328-server [ 1575.375569] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 01:16:54 (1787116614) [ 1576.840346] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg328-server [ 1578.563710] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 01:16:57 (1787116617) [ 1580.283549] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg328-server [ 1582.171516] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 01:17:00 (1787116620) [ 1582.268964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1582.286481] LNet: 82152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1582.301614] LNetError: 82152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1582.315471] LNet: Removed LNI 192.168.203.28@tcp [ 1583.336738] LNet: Removed LNI 192.168.203.28@tcp1 [ 1583.369801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1583.483737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1583.506376] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1584.191809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1584.293486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 1584.536362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1584.589675] LNet: 82306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1584.597346] LNetError: 82306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1584.604324] LNetError: 82306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1584.630432] LNet: Removed LNI 192.168.203.28@tcp [ 1584.649205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1584.703498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1584.740625] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1585.524740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 1585.696418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1585.720800] LNetError: 82452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1586.794906] LNet: Removed LNI 192.168.203.28@tcp [ 1586.831307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1586.945585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1586.968587] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1587.602581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1587.681596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 1587.894805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1587.908993] LNet: 82599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1587.919309] LNet: 82599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1587.929021] LNetError: 82599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1587.945891] LNet: Removed LNI 192.168.203.28@tcp [ 1587.975161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1588.022519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1588.058414] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1588.851343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 1589.069485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1590.121155] LNet: Removed LNI 192.168.203.28@tcp [ 1590.174923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1590.257805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1590.291838] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1591.133044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1591.245435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 1591.467699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1592.549771] LNet: Removed LNI 192.168.203.28@tcp [ 1592.600269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1592.659714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1592.685303] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1593.561634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 1593.766554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1593.781211] LNet: 83037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1593.785051] LNet: 83037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1593.790092] LNetError: 83037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1593.794940] LNetError: 83037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1593.822724] LNet: Removed LNI 192.168.203.28@tcp [ 1593.849957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1593.904882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1593.926423] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1594.611970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1594.704914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 1594.997236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1596.081346] LNet: Removed LNI 192.168.203.28@tcp [ 1596.127377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1596.199090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1596.240308] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1597.095895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 1599.554295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1600.614727] LNet: Removed LNI 192.168.203.28@tcp [ 1600.654986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1600.722991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1600.760491] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1601.385531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1601.484963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 1602.909396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1602.926925] LNet: 83505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1602.934615] LNet: 83505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1602.940738] LNetError: 83505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1602.950567] LNetError: 83505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1602.972958] LNet: Removed LNI 192.168.203.28@tcp [ 1603.005943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1603.072072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1603.103874] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1603.977800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 1604.210821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1605.290324] LNet: Removed LNI 192.168.203.28@tcp [ 1605.330515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1605.395829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1605.434369] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1606.219884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1606.329178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 1606.623837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1607.654847] LNet: Removed LNI 192.168.203.28@tcp [ 1607.710677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1607.774695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1607.796615] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1608.640747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 1610.072767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1611.115301] LNet: Removed LNI 192.168.203.28@tcp [ 1611.160792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1611.231508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1611.266617] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1612.101173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1612.177047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1612.212103] LNet: Removed LNI 192.168.203.28@tcp [ 1612.233460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1612.297067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1612.331382] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1612.947138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1613.005984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1613.027821] LNet: Removed LNI 192.168.203.28@tcp [ 1613.049424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1613.119572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1613.296898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1613.349497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1613.398995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1613.543829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1613.597806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1613.697828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1613.797572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1613.861797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1613.948350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1614.014595] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1615.043964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1615.085547] LNet: Removed LNI 192.168.203.28@tcp [ 1615.120163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1615.229395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1622.824869] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 01:17:41 (1787116661) [ 1623.095214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1623.120302] LNet: 84974:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1623.124459] LNet: 84974:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1623.141434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1623.192458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1623.225606] LNet: Added LNI 192.168.203.28@tcp231 [8/256/0/180] [ 1624.561787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1624.579842] LNetError: 85112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1624.586432] LNetError: 85112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1624.605467] LNet: Removed LNI 192.168.203.28@tcp231 [ 1624.620283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1624.697946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1624.722510] LNet: Added LNI 192.168.203.28@tcp231 [8/256/0/180] [ 1624.728855] LNet: 85120:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1625.239722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1625.285207] LNet: Removed LNI 192.168.203.28@tcp231 [ 1625.302261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1625.346996] LNet: Added LNI 192.168.203.28@tcp231 [8/256/0/180] [ 1626.851789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1626.893966] LNet: Removed LNI 192.168.203.28@tcp231 [ 1626.924598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1627.008557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1627.031543] LNet: Added LNI 192.168.203.28@tcp231 [8/256/0/0] [ 1627.037798] LNet: 85273:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1627.469215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1627.500592] LNet: Removed LNI 192.168.203.28@tcp231 [ 1627.523355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1627.557296] LNet: Added LNI 192.168.203.28@tcp231 [8/256/0/0] [ 1628.648865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1628.682509] LNet: Removed LNI 192.168.203.28@tcp231 [ 1628.719614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1628.809045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1628.839861] LNet: Added LNI 192.168.203.28@tcp231 [8/256/0/60] [ 1628.849374] LNet: 85424:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1629.523202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1629.546668] LNet: Removed LNI 192.168.203.28@tcp231 [ 1629.569405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1629.619729] LNet: Added LNI 192.168.203.28@tcp231 [8/256/0/60] [ 1631.233464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1631.283200] LNet: Removed LNI 192.168.203.28@tcp231 [ 1631.314928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1631.385689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1631.436493] LNet: Added LNI 192.168.203.28@tcp231 [8/256/0/180] [ 1631.450177] LNet: 85576:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1632.050234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1632.126604] LNet: Removed LNI 192.168.203.28@tcp231 [ 1632.166739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1632.210482] LNet: Added LNI 192.168.203.28@tcp231 [8/256/0/180] [ 1641.109646] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 01:17:59 (1787116679) [ 1641.276592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1641.333911] LNet: Removed LNI 192.168.203.28@tcp231 [ 1641.367803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1641.416229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1641.441615] LNet: Added LNI 192.168.203.28@tcp232 [8/256/0/180] [ 1642.100656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1642.154032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1642.188605] LNet: Removed LNI 192.168.203.28@tcp232 [ 1642.216506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1642.254697] LNet: Added LNI 192.168.203.28@tcp232 [8/256/0/180] [ 1643.074565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1643.104852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1643.133156] LNet: Removed LNI 192.168.203.28@tcp232 [ 1643.151568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1643.205538] LNet: Added LNI 192.168.203.28@tcp232 [8/256/0/180] [ 1643.838967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1643.886754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1643.919658] LNet: Removed LNI 192.168.203.28@tcp232 [ 1643.938964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1643.964488] LNet: Added LNI 192.168.203.28@tcp232 [8/256/0/180] [ 1644.586757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1644.625709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1644.661296] LNet: Removed LNI 192.168.203.28@tcp232 [ 1644.677570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1650.653399] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 01:18:09 (1787116689) [ 1651.918671] Key type .llcrypt unregistered [ 1651.921143] Key type ._llcrypt unregistered [ 1658.665838] Key type ._llcrypt registered [ 1658.669590] Key type .llcrypt registered [ 1658.861911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1658.877731] alg: No test for adler32 (adler32-zlib) [ 1659.801882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1659.843528] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1666.483666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.128@tcp [ 1674.915364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1674.969699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1674.997875] LNet: Added LNI 192.168.203.28@tcp1 [8/256/0/180] [ 1676.563372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e local_interrupt -r 2 [ 1676.630222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp1 -m GET -e local_interrupt -r 2 [ 1676.689413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e local_interrupt -r 2 [ 1676.744662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp1 -m GET -e local_interrupt -r 2 [ 1676.802078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp -m GET -e local_interrupt -r 2 [ 1676.877871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp1 -m GET -e local_interrupt -r 2 [ 1676.977665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp -m GET -e local_interrupt -r 2 [ 1677.052903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp1 -m GET -e local_interrupt -r 2 [ 1677.098747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.28@tcp 192.168.203.128@tcp [ 1677.120078] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1677.129093] LNet: 1 local NIs in recovery (showing 1): 192.168.203.28@tcp [ 1677.379896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1677.412409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1677.457648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1677.486964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1677.521905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1677.553569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1677.581624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1677.625195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1677.651695] LNet: 88076:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1677.669156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1677.718882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1677.758642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1677.784621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1677.825712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1677.860466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1678.325575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e local_dropped -r 2 [ 1678.350980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp1 -m GET -e local_dropped -r 2 [ 1678.378080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e local_dropped -r 2 [ 1678.418508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp1 -m GET -e local_dropped -r 2 [ 1678.475365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp -m GET -e local_dropped -r 2 [ 1678.509677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp1 -m GET -e local_dropped -r 2 [ 1678.570163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp -m GET -e local_dropped -r 2 [ 1678.613672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp1 -m GET -e local_dropped -r 2 [ 1678.658809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.28@tcp 192.168.203.128@tcp [ 1678.679532] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1678.938992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1678.977405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1679.000896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1679.023370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1679.050809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1679.091741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1679.123842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1679.155720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1679.178910] LNet: 88196:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1679.201687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1679.233808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1679.277634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1679.316774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1679.357462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1679.411534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1680.041374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e local_aborted -r 2 [ 1680.072837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp1 -m GET -e local_aborted -r 2 [ 1680.107583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e local_aborted -r 2 [ 1680.142918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp1 -m GET -e local_aborted -r 2 [ 1680.179583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp -m GET -e local_aborted -r 2 [ 1680.213628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp1 -m GET -e local_aborted -r 2 [ 1680.242756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp -m GET -e local_aborted -r 2 [ 1680.273210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp1 -m GET -e local_aborted -r 2 [ 1680.305233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.28@tcp 192.168.203.128@tcp [ 1680.324116] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1680.533735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1680.582585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1680.630412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1680.661609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1680.688756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1680.718458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1680.748618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1680.778470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1680.797288] LNet: 88316:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1680.821634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1680.853470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1680.885622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1680.922420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1680.951447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1680.988497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1681.537859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e local_no_route -r 2 [ 1681.577744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp1 -m GET -e local_no_route -r 2 [ 1681.625704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e local_no_route -r 2 [ 1681.672277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp1 -m GET -e local_no_route -r 2 [ 1681.752354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp -m GET -e local_no_route -r 2 [ 1681.806781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp1 -m GET -e local_no_route -r 2 [ 1681.854680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp -m GET -e local_no_route -r 2 [ 1681.894886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp1 -m GET -e local_no_route -r 2 [ 1681.945421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.28@tcp 192.168.203.128@tcp [ 1682.175554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1682.208338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1682.237636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1682.268608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1682.293987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1682.319208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1682.353434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1682.402345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1682.453157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1682.533659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1682.568870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1682.600796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1682.648374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1682.689491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1683.291327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e local_timeout -r 2 [ 1683.346826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp1 -m GET -e local_timeout -r 2 [ 1683.396359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e local_timeout -r 2 [ 1683.430298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp1 -m GET -e local_timeout -r 2 [ 1683.464570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp -m GET -e local_timeout -r 2 [ 1683.529692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp1 -m GET -e local_timeout -r 2 [ 1683.624533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp -m GET -e local_timeout -r 2 [ 1683.700311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp1 -m GET -e local_timeout -r 2 [ 1683.741659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.28@tcp 192.168.203.128@tcp [ 1683.767636] LNet: There was an unexpected network error while writing to 192.168.203.128: rc = -22 [ 1683.776858] LNet: Skipped 1 previous similar message [ 1684.071366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1684.116584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1684.159618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1684.192296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1684.222438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1684.250756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1684.309560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1684.338359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1684.366294] LNet: 88557:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1684.376679] LNet: 88557:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1684.402430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1684.437940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1684.478685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1684.503206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1684.523497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1684.556263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1685.116708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp -m GET -e remote_dropped -r 2 [ 1685.163415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.128@tcp1 -m GET -e remote_dropped -r 2 [ 1685.231559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp -m GET -e remote_dropped -r 2 [ 1685.299737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp -d 192.168.203.28@tcp1 -m GET -e remote_dropped -r 2 [ 1685.350359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp -m GET -e remote_dropped -r 2 [ 1685.395528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.28@tcp1 -d 192.168.203.128@tcp1 -m GET -e remote_dropped -r 2 [ 1685.463145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp -m GET -e remote_dropped -r 2 [ 1685.494148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.28@tcp1 -d 192.168.203.28@tcp1 -m GET -e remote_dropped -r 2 [ 1685.528522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.28@tcp 192.168.203.128@tcp [ 1685.545967] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.128@tcp [ 1685.782622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1685.821219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1685.862340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1685.895856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1685.930828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1685.964732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1686.000248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1686.034486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1686.061422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1686.106133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1686.153347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1686.179381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1686.213694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1686.259330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1691.824881] LNet: 88751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1691.834319] LNetError: 88751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1691.839893] LNet: Removed LNI 192.168.203.28@tcp [ 1691.860565] LNet: Removed LNI 192.168.203.28@tcp1 [ 1692.321078] Key type .llcrypt unregistered [ 1692.328793] Key type ._llcrypt unregistered [ 1710.434257] Key type ._llcrypt registered [ 1710.441874] Key type .llcrypt registered [ 1711.917054] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 01:19:10 (1787116750) [ 1712.491224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1712.501842] alg: No test for adler32 (adler32-zlib) [ 1713.443398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1713.508302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1713.563792] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1714.563300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1714.614198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.28@tcp [ 1714.661040] LNetError: 89730:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.28@tcp to recovery [ 1714.772250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.28@tcp [ 1714.887741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1721.633814] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 01:19:20 (1787116760) [ 1721.871986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1721.912256] LNet: 90322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1721.921627] LNetError: 90322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1721.948747] LNet: Removed LNI 192.168.203.28@tcp [ 1721.982188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1722.047870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1722.085049] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1723.067600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1723.107427] LNet: Added LNI 192.168.203.28@tcp1 [8/256/0/180] [ 1723.979226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.28@tcp [ 1724.004745] LNetError: 90588:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.28@tcp to recovery [ 1724.079603] LNet: 1 local NIs in recovery (showing 1): 192.168.203.28@tcp [ 1724.081146] LNet: There was an unexpected network error while writing to 192.168.203.28: rc = -22 [ 1725.261868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.28@tcp [ 1733.822578] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 01:19:32 (1787116772) [ 1736.080540] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg328-server [ 1738.124895] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 01:19:36 (1787116776) [ 1740.091520] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg328-server [ 1742.235737] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 01:19:40 (1787116780) [ 1744.053524] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg328-server [ 1746.203496] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 01:19:44 (1787116784) [ 1746.603704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1746.633933] LNet: 92210:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1746.662392] LNetError: 92210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1746.692165] LNet: Removed LNI 192.168.203.28@tcp [ 1746.710048] LNet: Removed LNI 192.168.203.28@tcp1 [ 1746.749875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1746.805180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1746.830805] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1747.616347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1747.691695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1747.759338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1747.838716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1747.876646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1747.900452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1754.975443] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 01:19:53 (1787116793) [ 1755.100719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1755.126856] LNet: 92930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1755.136474] LNetError: 92930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1755.147099] LNetError: 92930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1755.167773] LNet: Removed LNI 192.168.203.28@tcp [ 1755.225353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1755.281946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1755.321178] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1764.208576] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 01:20:02 (1787116802) [ 1764.339788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1764.366513] LNet: 93632:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1764.375916] LNetError: 93632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1764.399574] LNet: Removed LNI 192.168.203.28@tcp [ 1764.425283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1764.517507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1764.537386] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1772.986845] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 01:20:11 (1787116811) [ 1774.406653] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1776.182705] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 01:20:15 (1787116815) [ 1776.577329] LNet: 94673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1776.581687] LNetError: 94673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1776.596786] LNet: Removed LNI 192.168.203.28@tcp [ 1777.206823] Key type .llcrypt unregistered [ 1777.208809] Key type ._llcrypt unregistered [ 1782.397390] Key type ._llcrypt registered [ 1782.400479] Key type .llcrypt registered [ 1782.598697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1782.613932] alg: No test for adler32 (adler32-zlib) [ 1783.543911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1783.578624] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1791.032774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.128@tcp [ 1795.476977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.128@tcp [ 1795.898800] LNet: 95090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1795.914383] LNetError: 95090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1795.931825] LNet: Removed LNI 192.168.203.28@tcp [ 1796.545608] Key type .llcrypt unregistered [ 1796.547456] Key type ._llcrypt unregistered [ 1808.577962] Key type ._llcrypt registered [ 1808.580436] Key type .llcrypt registered [ 1809.941808] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1811.770463] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1813.258233] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 01:20:52 (1787116852) [ 1814.825426] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1816.365287] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 01:20:55 (1787116855) [ 1818.132447] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg328-server [ 1819.791891] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 01:20:58 (1787116858) [ 1821.694569] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg328-server [ 1823.666985] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 01:21:02 (1787116862) [ 1824.943719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1824.966326] alg: No test for adler32 (adler32-zlib) [ 1826.080821] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 1826.391425] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1828.071154] Key type lgssc registered [ 1829.221191] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1849.657312] Key type lgssc unregistered [ 1849.941697] LNet: 98266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1849.953209] LNetError: 98266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1849.985962] LNet: Removed LNI 192.168.203.28@tcp [ 1850.757712] Key type .llcrypt unregistered [ 1850.760501] Key type ._llcrypt unregistered [ 1867.847090] Key type ._llcrypt registered [ 1867.849667] Key type .llcrypt registered [ 1869.887262] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 01:21:47 (1787116907) [ 1870.486611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1870.517886] alg: No test for adler32 (adler32-zlib) [ 1871.441309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1871.530563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1871.573267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1871.601750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1871.644947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1879.592154] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 01:21:58 (1787116918) [ 1880.033833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1880.090580] LNet: 99681:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1880.142372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1880.236175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1880.270498] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1881.075466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1881.087195] LNet: 99826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1881.090590] LNetError: 99826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1881.109096] LNet: Removed LNI 192.168.203.28@tcp [ 1881.123281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1881.157727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1881.177673] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1881.623569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1881.664848] LNetError: 99854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1881.689622] LNet: Removed LNI 192.168.203.28@tcp [ 1881.719145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1881.767705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1881.798925] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1890.122500] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 01:22:08 (1787116928) [ 1890.323978] LNet: 100444:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1890.330707] LNet: 100444:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1890.336874] LNetError: 100444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1890.364934] LNet: Removed LNI 192.168.203.28@tcp [ 1890.939772] Key type .llcrypt unregistered [ 1890.941656] Key type ._llcrypt unregistered [ 1898.613383] Key type ._llcrypt registered [ 1898.624483] Key type .llcrypt registered [ 1898.907533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1898.930247] alg: No test for adler32 (adler32-zlib) [ 1899.967858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1900.110484] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1900.113760] LNet: *** cfs_fail_loc=e003, val=100*** [ 1900.865440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 1900.893259] LNet: *** cfs_fail_loc=e003, val=100*** [ 1900.897635] LNet: Skipped 2 previous similar messages [ 1900.905475] Ping reply received with mapping: [ 1900.908887] found metadata reply NI[192.168.203.28@tcp] [ 1900.912940] 1c cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1900.917499] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1900.921991] 5a 5a 5a 5a 5a 5a 5a 5a [ 1901.044664] LNet: 100842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1901.051690] LNetError: 100842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1901.070913] LNet: Removed LNI 192.168.203.28@tcp [ 1901.816863] Key type .llcrypt unregistered [ 1901.820083] Key type ._llcrypt unregistered [ 1913.578903] Key type ._llcrypt registered [ 1913.582284] Key type .llcrypt registered [ 1914.922281] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 01:22:33 (1787116953) [ 1915.364285] Key type .llcrypt unregistered [ 1915.366496] Key type ._llcrypt unregistered [ 1921.493107] Key type ._llcrypt registered [ 1921.501853] Key type .llcrypt registered [ 1921.914795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1921.945556] alg: No test for adler32 (adler32-zlib) [ 1923.057113] LNetError: 101887:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1923.642255] Key type .llcrypt unregistered [ 1923.650901] Key type ._llcrypt unregistered [ 1929.926107] Key type ._llcrypt registered [ 1929.927836] Key type .llcrypt registered [ 1931.626541] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 01:22:49 (1787116969) [ 1932.310036] Key type .llcrypt unregistered [ 1932.315898] Key type ._llcrypt unregistered [ 1942.334544] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1942.342140] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1945.191553] Key type ._llcrypt registered [ 1945.194819] Key type .llcrypt registered [ 1945.433054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1945.454197] alg: No test for adler32 (adler32-zlib) [ 1946.570990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1946.662960] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1958.496197] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 01:23:17 (1787116997) [ 1958.591169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1958.601735] LNet: 103661:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1958.606442] LNetError: 103661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1958.616293] LNet: Removed LNI 10.1.2.3@tcp [ 1958.636730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1959.388641] Key type .llcrypt unregistered [ 1959.394902] Key type ._llcrypt unregistered [ 1966.104668] Key type ._llcrypt registered [ 1966.108059] Key type .llcrypt registered [ 1966.368249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1966.406210] alg: No test for adler32 (adler32-zlib) [ 1967.380233] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1967.477305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 1967.629730] LNet: 103957:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1967.635811] LNetError: 103957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1967.651588] LNet: Removed LNI 192.168.203.28@tcp [ 1968.529403] Key type .llcrypt unregistered [ 1968.534154] Key type ._llcrypt unregistered [ 1979.897305] Key type ._llcrypt registered [ 1979.902169] Key type .llcrypt registered [ 1981.120874] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 01:23:39 (1787117019) [ 1981.736438] Key type .llcrypt unregistered [ 1981.738301] Key type ._llcrypt unregistered [ 1988.174293] Key type ._llcrypt registered [ 1988.176345] Key type .llcrypt registered [ 1988.365961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1988.390657] alg: No test for adler32 (adler32-zlib) [ 1996.597383] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 01:23:55 (1787117035) [ 1996.681063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1996.730862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1996.799133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1996.833766] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 1997.697931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1997.731409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1997.783626] LNet: Removed LNI 192.168.203.28@tcp [ 1998.081826] LNet: 105715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1998.617420] Key type .llcrypt unregistered [ 1998.620024] Key type ._llcrypt unregistered [ 2010.406509] Key type ._llcrypt registered [ 2010.409527] Key type .llcrypt registered [ 2011.812408] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 01:24:10 (1787117050) [ 2012.243490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2012.268162] alg: No test for adler32 (adler32-zlib) [ 2013.175371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2013.238670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2013.269333] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2014.059219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 2014.110453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.28@tcp [ 2014.132881] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2014.138963] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2014.141728] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2014, 0, 0 [ 2014.151090] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2014.160109] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2014.166277] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2014.169726] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2014, 0, 0 [ 2014.176183] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2014.181936] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2014.186871] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2014.195806] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2014, 0, 2064 [ 2014.204798] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2014.207646] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2014.210797] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2014.215208] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2014, 0, 2064 [ 2014.219666] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2020.466896] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 01:24:19 (1787117059) [ 2021.031265] LNet: 107256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2021.037436] LNetError: 107256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2021.072047] LNet: Removed LNI 192.168.203.28@tcp [ 2021.719404] Key type .llcrypt unregistered [ 2021.722515] Key type ._llcrypt unregistered [ 2027.931775] Key type ._llcrypt registered [ 2027.934124] Key type .llcrypt registered [ 2028.268057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2028.282887] alg: No test for adler32 (adler32-zlib) [ 2029.222399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2029.264634] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2037.509961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.128@tcp [ 2047.323195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2047.347298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2047.376447] LNet: Added LNI 192.168.203.28@tcp1 [8/256/0/180] [ 2048.333911] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2048.402660] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2048.408499] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2048.415328] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2050.767410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2050.800622] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2062.450508] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 01:25:01 (1787117101) [ 2062.616879] LNet: 108756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2062.649785] LNetError: 108756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2062.675049] LNet: Removed LNI 192.168.203.28@tcp [ 2063.711378] LNet: Removed LNI 192.168.203.28@tcp1 [ 2063.714079] LNetError: 108756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2063.720673] LNetError: 108756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2063.743847] LNet: Removed LNI 10.1.2.3@tcp99 [ 2064.591803] Key type .llcrypt unregistered [ 2064.593835] Key type ._llcrypt unregistered [ 2070.352163] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2070.355891] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2073.018511] Key type ._llcrypt registered [ 2073.021854] Key type .llcrypt registered [ 2073.316711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2073.335726] alg: No test for adler32 (adler32-zlib) [ 2074.419235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2074.498992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2074.539886] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2075.646399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2075.685073] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2078.645609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2078.741578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2078.810894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2078.870176] LNetError: 109509:0:(peer.c:2277:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2078.926184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2078.989184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 2079.375682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2079.439626] LNet: 109529:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2079.448528] LNetError: 109529:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2079.454219] LNet: Removed LNI 10.1.2.3@tcp [ 2080.485578] LNet: Removed LNI 192.168.203.28@tcp [ 2081.167506] Key type .llcrypt unregistered [ 2081.170340] Key type ._llcrypt unregistered [ 2093.712518] Key type ._llcrypt registered [ 2093.716822] Key type .llcrypt registered [ 2095.166442] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 01:25:33 (1787117133) [ 2095.771576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2095.799405] alg: No test for adler32 (adler32-zlib) [ 2096.774651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2096.839542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2096.872754] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2097.675550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 2097.771149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg328-client@tcp [ 2103.866869] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 01:25:42 (1787117142) [ 2104.208332] LNet: 111062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2104.218132] LNetError: 111062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2104.235596] LNet: Removed LNI 192.168.203.28@tcp [ 2104.932967] Key type .llcrypt unregistered [ 2104.934745] Key type ._llcrypt unregistered [ 2111.603750] Key type ._llcrypt registered [ 2111.610480] Key type .llcrypt registered [ 2111.844918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2111.868245] alg: No test for adler32 (adler32-zlib) [ 2112.785803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2112.804247] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2112.828780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2119.479926] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 01:25:58 (1787117158) [ 2121.260429] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2123.387286] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 01:26:01 (1787117161) [ 2125.777670] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2128.049732] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 01:26:06 (1787117166) [ 2128.169046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2128.233598] LNet: 112475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2128.246935] LNetError: 112475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2128.270423] LNet: Removed LNI 192.168.203.28@tcp [ 2128.307393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2128.357546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2128.378199] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2129.374725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2129.472670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2138.153645] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 01:26:16 (1787117176) [ 2138.333907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2138.392824] LNet: 113188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2138.417665] LNetError: 113188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2138.485043] LNet: Removed LNI 192.168.203.28@tcp [ 2138.580541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2138.691644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2138.763375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2139.711610] Key type .llcrypt unregistered [ 2139.715412] Key type ._llcrypt unregistered [ 2153.180034] Key type ._llcrypt registered [ 2153.185286] Key type .llcrypt registered [ 2154.413985] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 01:26:33 (1787117193) [ 2155.009852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2155.025942] alg: No test for adler32 (adler32-zlib) [ 2156.009181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2156.102461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2156.152919] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2156.724696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2156.774935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 2162.980878] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 01:26:41 (1787117201) [ 2163.067799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2163.087158] LNet: 114744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2163.093630] LNetError: 114744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2164.135049] LNet: Removed LNI 192.168.203.28@tcp [ 2164.173951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2164.246285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2164.270909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2170.558365] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 01:26:49 (1787117209) [ 2170.796950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2170.827537] LNet: 115329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2170.855960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2170.937264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2171.381483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2171.434163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2171.535947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2171.870404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2171.915397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2171.993454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2172.018896] LNet: 115362:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2172.058287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2172.152398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2172.495533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2172.548674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2172.642247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2173.119510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2173.165312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2179.751876] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 01:26:58 (1787117218) [ 2180.076699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2180.110141] LNet: 115962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2180.151136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2180.223921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2180.248702] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2181.071492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2181.096227] LNet: Added LNI 192.168.203.28@tcp2 [8/256/0/180] [ 2181.977737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2182.084775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 2182.353836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2182.383839] LNetError: 116241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2183.391262] LNet: Removed LNI 192.168.203.28@tcp [ 2183.397259] LNetError: 116241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2183.412851] LNet: Removed LNI 192.168.203.28@tcp2 [ 2183.458493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2183.539403] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2183.547672] LNet: Added LNI 192.168.203.28@tcp2 [8/256/0/180] [ 2183.552178] LNet: 116248:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2184.177563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 2184.451561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2184.488928] LNet: 116285:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2184.493090] LNet: 116285:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2185.567201] LNet: Removed LNI 192.168.203.28@tcp [ 2185.574929] LNetError: 116285:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2185.589133] LNetError: 116285:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2185.621353] LNet: Removed LNI 192.168.203.28@tcp2 [ 2185.660279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2185.739501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2185.776357] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2186.395877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2186.412876] LNet: Added LNI 192.168.203.28@tcp2 [8/256/0/180] [ 2187.335858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 2187.383825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2187.426498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 2187.640453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2188.703721] LNet: Removed LNI 192.168.203.28@tcp [ 2189.736717] LNet: Removed LNI 192.168.203.28@tcp2 [ 2189.758980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2189.833436] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2189.842456] LNet: Added LNI 192.168.203.28@tcp2 [8/256/0/180] [ 2189.850228] LNet: 116575:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2190.306992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 2190.590079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2190.617943] LNetError: 116613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2190.624234] LNetError: 116613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2190.630470] LNet: Removed LNI 192.168.203.28@tcp [ 2191.653965] LNet: Removed LNI 192.168.203.28@tcp2 [ 2191.711218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2191.788287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2191.840855] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2192.471777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2192.491573] LNet: Added LNI 192.168.203.28@tcp2 [8/256/0/180] [ 2193.272402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2193.348171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 2193.559883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2193.575595] LNet: 116892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2193.579896] LNet: 116892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2194.591132] LNet: Removed LNI 192.168.203.28@tcp [ 2194.603313] LNet: Removed LNI 192.168.203.28@tcp2 [ 2194.627608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2194.706222] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2194.713250] LNet: Added LNI 192.168.203.28@tcp2 [8/256/0/180] [ 2194.721462] LNet: 116898:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2195.210340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 2195.453290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2196.511190] LNet: Removed LNI 192.168.203.28@tcp [ 2196.526603] LNet: Removed LNI 192.168.203.28@tcp2 [ 2196.567790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2196.659958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2196.690812] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2197.477223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2197.494329] LNet: Added LNI 192.168.203.28@tcp2 [8/256/0/180] [ 2198.527234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 2198.582796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2198.665529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 2198.885280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2198.903771] LNetError: 117220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2198.914903] LNetError: 117220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2198.926210] LNet: Removed LNI 192.168.203.28@tcp [ 2199.986047] LNet: Removed LNI 192.168.203.28@tcp2 [ 2200.023948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2200.088349] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2200.097302] LNet: Added LNI 192.168.203.28@tcp2 [8/256/0/180] [ 2200.104845] LNet: 117226:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2200.766415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 2200.994424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2201.023830] LNet: Removed LNI 192.168.203.28@tcp [ 2202.085572] LNet: Removed LNI 192.168.203.28@tcp2 [ 2202.148701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2202.222412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2202.251863] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2203.149321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2203.182218] LNet: Added LNI 192.168.203.28@tcp2 [8/256/0/180] [ 2203.836567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2203.865771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2203.924715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 2204.278294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2205.343305] LNet: Removed LNI 192.168.203.28@tcp [ 2205.357789] LNet: Removed LNI 192.168.203.28@tcp2 [ 2205.394232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2205.454260] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2205.459853] LNet: Added LNI 192.168.203.28@tcp2 [8/256/0/180] [ 2205.465154] LNet: 117566:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2206.048366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 2206.366996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2207.462342] LNet: Removed LNI 192.168.203.28@tcp [ 2207.476735] LNet: Removed LNI 192.168.203.28@tcp2 [ 2207.514977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2207.557546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2207.589291] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2208.249394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2208.262912] LNet: Added LNI 192.168.203.28@tcp2 [8/256/0/180] [ 2209.140716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 2209.170451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2209.213469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2209.269058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 2209.685340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2209.712310] LNet: 117919:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2209.725961] LNet: 117919:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2210.783169] LNet: Removed LNI 192.168.203.28@tcp [ 2211.814884] LNet: Removed LNI 192.168.203.28@tcp2 [ 2211.827848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2211.865551] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2211.870324] LNet: Added LNI 192.168.203.28@tcp2 [8/256/0/180] [ 2212.211398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 2221.100634] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2223.036728] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 01:27:41 (1787117261) [ 2223.245393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2223.287252] LNetError: 118680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2223.298935] LNetError: 118680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 2223.311520] LNet: Removed LNI 192.168.203.28@tcp [ 2224.359607] LNet: Removed LNI 192.168.203.28@tcp2 [ 2224.465666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2224.931903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2224.976399] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2225.906464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2225.921027] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2226.960200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2227.024981] LNet: Removed LNI 10.1.2.3@tcp [ 2227.049282] LNet: Removed LNI 192.168.203.28@tcp [ 2227.094708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2227.198971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2227.245642] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2227.254617] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2227.898929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2227.939866] LNet: Removed LNI 192.168.203.28@tcp [ 2227.954808] LNet: Removed LNI 10.1.2.3@tcp [ 2235.330619] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 01:27:54 (1787117274) [ 2235.504118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2235.555328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2235.615202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2235.636801] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2236.463455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2236.520728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 2236.564125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 2236.597473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 2236.617364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 2236.645842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 2236.664978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 2236.688846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 2236.716050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 2236.739295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 2236.768171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.28@tcp [ 2236.879182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2244.184435] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 01:28:02 (1787117282) [ 2244.450243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2244.503810] LNet: 120296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2244.512674] LNet: 120296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2245.543379] LNet: Removed LNI 192.168.203.28@tcp [ 2245.608752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2245.812887] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2246.051091] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2246.053747] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2246.068619] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2248.463349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2248.516078] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2249.414490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2249.432847] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2251.564292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.28@tcp [ 2251.619473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2251.690891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.28@tcp -m PUT -e local_timeout -r 1 [ 2251.817704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2251.873628] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2251.885278] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2252.066731] LNetError: 120358:0:(lib-move.c:4801:lnet_parse()) 192.168.203.28@tcp, src 192.168.203.28@tcp: Dropping PUT (error -108 looking up sender) [ 2253.092159] LNet: Removed LNI 10.1.2.3@tcp [ 2254.128172] LNet: Removed LNI 192.168.203.28@tcp [ 2254.702609] Key type .llcrypt unregistered [ 2254.705488] Key type ._llcrypt unregistered [ 2268.785834] Key type ._llcrypt registered [ 2268.789599] Key type .llcrypt registered [ 2270.434887] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 01:28:28 (1787117308) [ 2270.867530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2270.887439] alg: No test for adler32 (adler32-zlib) [ 2271.883685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2272.347276] LNet: 121669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2273.049506] Key type .llcrypt unregistered [ 2273.051834] Key type ._llcrypt unregistered [ 2280.380352] Key type ._llcrypt registered [ 2280.382746] Key type .llcrypt registered [ 2280.761682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2280.789353] alg: No test for adler32 (adler32-zlib) [ 2281.711400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2281.764726] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2291.405289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.128@tcp [ 2291.810867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2292.838762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.128@tcp [ 2293.089019] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.128@tcp [ 2293.153790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2293.194602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2293.222994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2293.254289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2293.289841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2293.320390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2293.358467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2293.392445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2293.427174] LNet: 122119:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2293.451950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2293.501625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2293.536172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2293.592194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2293.652949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2293.692905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2294.388886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2294.679508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2294.747553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2294.787385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2294.858537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2294.903432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2294.974880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2295.037410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2295.111741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2295.154488] LNet: 122236:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2295.191941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2295.230526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2295.261845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2295.295927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2295.352927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2295.407392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2296.435500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.128@tcp [ 2296.693831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2296.740384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2296.843754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2296.901567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2296.962478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2296.999795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2297.052554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2297.096463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2297.135409] LNet: 122354:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2297.161690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2297.199837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2297.239676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2297.291639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2297.333620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2297.388744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2298.139196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2298.556176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2298.606746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2298.648818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2298.705763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2298.764479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2298.806633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2298.844583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2298.886396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2298.910464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2298.942822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2298.970879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2299.006514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2299.044913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2299.111899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2299.472045] LNet: 122517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2299.483949] LNetError: 122517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2299.513400] LNet: Removed LNI 192.168.203.28@tcp [ 2300.054961] Key type .llcrypt unregistered [ 2300.063507] Key type ._llcrypt unregistered [ 2318.577805] Key type ._llcrypt registered [ 2318.579847] Key type .llcrypt registered [ 2319.828570] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 01:29:18 (1787117358) [ 2320.233069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2320.254854] alg: No test for adler32 (adler32-zlib) [ 2321.289637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2321.845282] LNet: 123404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2322.468801] Key type .llcrypt unregistered [ 2322.471235] Key type ._llcrypt unregistered [ 2328.973571] Key type ._llcrypt registered [ 2328.976609] Key type .llcrypt registered [ 2329.240967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2329.256520] alg: No test for adler32 (adler32-zlib) [ 2330.237252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2330.276608] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2338.427377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.128@tcp [ 2347.910990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2347.992225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2348.034185] LNet: Added LNI 192.168.203.28@tcp1 [8/256/0/180] [ 2349.416754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2350.409704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.128@tcp [ 2350.763376] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.128@tcp [ 2350.876425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2350.965649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2351.038739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2351.093205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2351.190703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2351.268325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2351.369946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2351.411442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2351.434992] LNet: 124056:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2351.481702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2351.569099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2351.620923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2351.672041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2351.723106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2351.769657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2353.029413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2353.362754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2353.424791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2353.473420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2353.530968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2353.579223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2353.641428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2353.752383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2353.797372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2353.821034] LNet: 124174:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2353.841822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2353.886764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2353.938390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2353.989664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2354.062863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2354.143680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2355.245768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.128@tcp [ 2355.607471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2355.645799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2355.700776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2355.761805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2355.820982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2355.914818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2355.971704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2356.028424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2356.074776] LNet: 124291:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2356.111752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2356.182986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2356.259538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2356.304811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2356.404269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2356.477947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2357.471951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2357.749326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2357.792865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2357.828861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2357.886595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2357.920806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2357.996381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2358.110609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2358.202510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2358.275124] LNet: 124409:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2358.349597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2358.418518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2358.453856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2358.512648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2358.587226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2358.656829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2366.327887] LNet: 124490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2366.337654] LNetError: 124490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2366.366386] LNet: Removed LNI 192.168.203.28@tcp [ 2366.380082] LNet: Removed LNI 192.168.203.28@tcp1 [ 2367.169245] Key type .llcrypt unregistered [ 2367.172422] Key type ._llcrypt unregistered [ 2387.185964] Key type ._llcrypt registered [ 2387.188479] Key type .llcrypt registered [ 2388.515724] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 01:30:27 (1787117427) [ 2390.023631] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2391.867829] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 01:30:30 (1787117430) [ 2399.419810] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 01:30:38 (1787117438) [ 2406.742873] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 01:30:45 (1787117445) [ 2414.344468] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 01:30:53 (1787117453) [ 2421.707499] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 01:31:00 (1787117460) [ 2429.018686] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 01:31:07 (1787117467) [ 2435.710519] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 01:31:14 (1787117474) [ 2442.485495] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 01:31:21 (1787117481) [ 2450.146587] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 01:31:28 (1787117488) [ 2457.148838] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 01:31:35 (1787117495) [ 2464.261239] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 01:31:42 (1787117502) [ 2472.835505] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 01:31:51 (1787117511) [ 2480.695666] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 01:31:59 (1787117519) [ 2490.794783] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 01:32:09 (1787117529) [ 2498.242954] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 01:32:17 (1787117537) [ 2506.118750] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 01:32:24 (1787117544) [ 2513.002641] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 01:32:31 (1787117551) [ 2521.165691] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 01:32:39 (1787117559) [ 2521.608320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2521.626598] alg: No test for adler32 (adler32-zlib) [ 2522.542166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2522.674207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2522.699880] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2523.917038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2523.936174] LNet: 135491:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2523.946425] LNetError: 135491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2523.961710] LNet: Removed LNI 192.168.203.28@tcp [ 2523.980902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2524.021783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2524.041785] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2530.831447] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 01:32:49 (1787117569) [ 2538.500134] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 01:32:57 (1787117577) [ 2546.797476] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 01:33:05 (1787117585) [ 2554.881777] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 01:33:13 (1787117593) [ 2562.348300] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 01:33:20 (1787117600) [ 2569.868634] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 01:33:28 (1787117608) [ 2578.140811] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 01:33:36 (1787117616) [ 2585.685663] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 01:33:44 (1787117624) [ 2594.754902] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 01:33:53 (1787117633) [ 2603.179481] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 01:34:01 (1787117641) [ 2610.008904] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 01:34:08 (1787117648) [ 2617.563434] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 01:34:16 (1787117656) [ 2624.918311] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2351 sec ========== 01:34:23 (1787117663) [ 2625.312253] LNet: 142956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2625.324335] LNetError: 142956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2625.337795] LNet: Removed LNI 192.168.203.28@tcp [ 2626.076573] Key type .llcrypt unregistered [ 2626.080632] Key type ._llcrypt unregistered [ 2638.513579] Key type ._llcrypt registered [ 2638.515286] Key type .llcrypt registered [ 2638.689650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2638.701716] alg: No test for adler32 (adler32-zlib) [ 2639.873848] Lustre: Lustre: Build Version: 2.17.56_4_gb718cad [ 2640.177542] LNet: Added LNI 192.168.203.28@tcp [8/256/0/180] [ 2641.959149] Key type lgssc registered [ 2643.841459] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2723.912055] Lustre: Mounted lustre-client [ 2729.693248] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2738.978390] Lustre: Unmounted lustre-client [ 2763.003606] Key type lgssc unregistered [ 2763.286642] LNet: 145886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2763.298854] LNetError: 145886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2763.336857] LNet: Removed LNI 192.168.203.28@tcp [ 2764.144161] Key type .llcrypt unregistered [ 2764.150043] Key type ._llcrypt unregistered