[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 401255152 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2421 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22BD 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 00227D (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2331 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23C1 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE23F9 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22bd-0xbffe2330] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22bc] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2331-0xbffe23c0] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23c1-0xbffe23f8] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe23f9-0xbffe2420] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003340] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009010] pid_max: default: 32768 minimum: 301 [ 0.010097] LSM: Security Framework initializing [ 0.011019] Yama: becoming mindful. [ 0.011678] SELinux: Initializing. [ 0.012042] *** VALIDATE selinux *** [ 0.020267] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025027] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027042] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028105] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029109] *** VALIDATE tmpfs *** [ 0.030454] *** VALIDATE proc *** [ 0.031254] *** VALIDATE cgroup *** [ 0.032010] *** VALIDATE cgroup2 *** [ 0.033286] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034152] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036030] Spectre V2 : User space: Vulnerable [ 0.037010] Speculative Store Bypass: Vulnerable [ 0.039807] debug: unmapping init [mem 0xffffffff8ce59000-0xffffffff8ce60fff] [ 0.041217] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042956] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044024] ... version: 2 [ 0.045015] ... bit width: 48 [ 0.046014] ... generic registers: 4 [ 0.047013] ... value mask: 0000ffffffffffff [ 0.048014] ... max period: 00007fffffffffff [ 0.049014] ... fixed-purpose events: 3 [ 0.050013] ... event mask: 000000070000000f [ 0.051291] rcu: Hierarchical SRCU implementation. [ 0.053331] smp: Bringing up secondary CPUs ... [ 0.054575] x86: Booting SMP configuration: [ 0.055021] .... node #0, CPUs: #1 #2 #3 [ 0.059094] smp: Brought up 1 node, 4 CPUs [ 0.061013] smpboot: Max logical packages: 1 [ 0.062012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.132061] node 0 deferred pages initialised in 68ms [ 0.135092] devtmpfs: initialized [ 0.136207] x86/mm: Memory block size: 128MB [ 0.138344] gcov: version magic: 0x41383552 [ 0.140206] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.143065] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.145363] pinctrl core: initialized pinctrl subsystem [ 0.147146] [ 0.147517] ************************************************************* [ 0.149008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.150007] ** ** [ 0.151006] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.153007] ** ** [ 0.155012] ** This means that this kernel is built to expose internal ** [ 0.156007] ** IOMMU data structures, which may compromise security on ** [ 0.158008] ** your system. ** [ 0.159008] ** ** [ 0.160006] ** If you see this message and you are not debugging the ** [ 0.162007] ** kernel, report this immediately to your vendor! ** [ 0.163006] ** ** [ 0.164009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165009] ************************************************************* [ 0.167568] NET: Registered protocol family 16 [ 0.168359] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.170038] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.172040] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.175012] cpuidle: using governor menu [ 0.176745] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.178356] PCI: Using configuration type 1 for base access [ 0.180129] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.188209] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.190018] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.194069] cryptd: max_cpu_qlen set to 1000 [ 0.196189] ACPI: Added _OSI(Module Device) [ 0.197012] ACPI: Added _OSI(Processor Device) [ 0.197992] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.199009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.202807] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.209296] ACPI: Interpreter enabled [ 0.210064] ACPI: PM: (supports S0 S3 S4 S5) [ 0.212014] ACPI: Using IOAPIC for interrupt routing [ 0.214088] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.216254] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.223789] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.225032] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.227021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.229067] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.231960] acpiphp: Slot [2] registered [ 0.233083] acpiphp: Slot [5] registered [ 0.234082] acpiphp: Slot [6] registered [ 0.234924] acpiphp: Slot [3] registered [ 0.235081] acpiphp: Slot [4] registered [ 0.235926] acpiphp: Slot [7] registered [ 0.237071] acpiphp: Slot [8] registered [ 0.237941] acpiphp: Slot [9] registered [ 0.239051] acpiphp: Slot [10] registered [ 0.239948] acpiphp: Slot [11] registered [ 0.240089] acpiphp: Slot [12] registered [ 0.240954] acpiphp: Slot [13] registered [ 0.242069] acpiphp: Slot [14] registered [ 0.242893] acpiphp: Slot [15] registered [ 0.244056] acpiphp: Slot [16] registered [ 0.244905] acpiphp: Slot [17] registered [ 0.246062] acpiphp: Slot [18] registered [ 0.246948] acpiphp: Slot [19] registered [ 0.247069] acpiphp: Slot [20] registered [ 0.248017] acpiphp: Slot [21] registered [ 0.248930] acpiphp: Slot [22] registered [ 0.250054] acpiphp: Slot [23] registered [ 0.250927] acpiphp: Slot [24] registered [ 0.252046] acpiphp: Slot [25] registered [ 0.253047] acpiphp: Slot [26] registered [ 0.253775] acpiphp: Slot [27] registered [ 0.254061] acpiphp: Slot [28] registered [ 0.254746] acpiphp: Slot [29] registered [ 0.255051] acpiphp: Slot [30] registered [ 0.257059] acpiphp: Slot [31] registered [ 0.257915] PCI host bridge to bus 0000:00 [ 0.258012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.259013] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.261013] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.262013] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.264013] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.265017] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.266189] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.267685] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.269820] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.275349] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.278008] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.279012] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.280008] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.281012] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.283413] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.284623] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.287039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.289668] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.293013] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.299012] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.303012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.307000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.314012] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.318013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.330027] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.338771] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.343013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.348016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.357014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.365079] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.367359] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.369288] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.371245] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.373141] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.376264] iommu: Default domain type: Passthrough [ 0.378264] SCSI subsystem initialized [ 0.379229] ACPI: bus type USB registered [ 0.381186] usbcore: registered new interface driver usbfs [ 0.383068] usbcore: registered new interface driver hub [ 0.384092] usbcore: registered new device driver usb [ 0.386161] pps_core: LinuxPPS API ver. 1 registered [ 0.388011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.391058] PTP clock support registered [ 0.393031] EDAC MC: Ver: 3.0.0 [ 0.394103] PCI: Using ACPI for IRQ routing [ 0.394874] NetLabel: Initializing [ 0.396007] NetLabel: domain hash size = 128 [ 0.396793] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.397055] NetLabel: unlabeled traffic allowed by default [ 0.399104] vgaarb: loaded [ 0.399547] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.399990] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.405019] clocksource: Switched to clocksource kvm-clock [ 0.503623] VFS: Disk quotas dquot_6.6.0 [ 0.505186] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.507920] *** VALIDATE ramfs *** [ 0.509224] *** VALIDATE hugetlbfs *** [ 0.510868] pnp: PnP ACPI init [ 0.513360] pnp: PnP ACPI: found 6 devices [ 0.529467] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.533137] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.534902] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.536206] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.538042] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.540441] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.542356] NET: Registered protocol family 2 [ 0.544828] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.548709] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.551842] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.556240] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.559342] TCP: Hash tables configured (established 65536 bind 65536) [ 0.561226] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.563112] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.564770] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.566592] NET: Registered protocol family 1 [ 0.568408] RPC: Registered named UNIX socket transport module. [ 0.569996] RPC: Registered udp transport module. [ 0.571349] RPC: Registered tcp transport module. [ 0.573112] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.575558] NET: Registered protocol family 44 [ 0.577164] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.578852] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.580470] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.581835] PCI: CLS 0 bytes, default 64 [ 0.582945] Unpacking initramfs... [ 1.943148] debug: unmapping init [mem 0xffff98277cc64000-0xffff98277ffcffff] [ 1.945924] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.947324] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.949353] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.402547] Initialise system trusted keyrings [ 2.403993] Key type blacklist registered [ 2.405481] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.411868] zbud: loaded [ 2.413992] *** VALIDATE nfs *** [ 2.415136] *** VALIDATE nfs4 *** [ 2.416220] pstore: using deflate compression [ 2.418826] Platform Keyring initialized [ 2.508428] NET: Registered protocol family 38 [ 2.510344] Key type asymmetric registered [ 2.511978] Asymmetric key parser 'x509' registered [ 2.513695] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.516929] io scheduler mq-deadline registered [ 2.518552] io scheduler kyber registered [ 2.520246] io scheduler bfq registered [ 2.522404] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.525641] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.528788] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.531938] ACPI: Power Button [PWRF] [ 2.539320] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.545596] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.561051] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.590193] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.618471] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.623350] Non-volatile memory driver v1.3 [ 2.625338] Linux agpgart interface v0.103 [ 2.654050] virtio_blk virtio1: [vda] 134024 512-byte logical blocks (68.6 MB/65.4 MiB) [ 2.657295] vda: detected capacity change from 0 to 68620288 [ 2.673514] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.676657] vdb: detected capacity change from 0 to 1073741824 [ 2.682917] libphy: Fixed MDIO Bus: probed [ 2.691954] usbcore: registered new interface driver usbserial_generic [ 2.694894] usbserial: USB Serial support registered for generic [ 2.698399] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.703196] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.705161] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.707397] mousedev: PS/2 mouse device common for all mice [ 2.711172] rtc_cmos 00:05: RTC can wake from S4 [ 2.714881] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.715772] rtc_cmos 00:05: registered as rtc0 [ 2.722745] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.722826] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.730087] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.732522] intel_pstate: CPU model not supported [ 2.739051] hid: raw HID events driver (C) Jiri Kosina [ 2.741157] usbcore: registered new interface driver usbhid [ 2.742914] usbhid: USB HID core driver [ 2.744121] drop_monitor: Initializing network drop monitor service [ 2.746150] Initializing XFRM netlink socket [ 2.747786] NET: Registered protocol family 10 [ 2.750292] Segment Routing with IPv6 [ 2.752234] NET: Registered protocol family 17 [ 2.753956] mpls_gso: MPLS GSO support [ 2.759863] RAS: Correctable Errors collector initialized. [ 2.762117] AVX version of gcm_enc/dec engaged. [ 2.763899] AES CTR mode by8 optimization enabled [ 2.824973] sched_clock: Marking stable (2824952591, 0)->(3655706557, -830753966) [ 2.828225] registered taskstats version 1 [ 2.831309] Loading compiled-in X.509 certificates [ 2.833108] zswap: loaded using pool lzo/zbud [ 2.858746] Key type big_key registered [ 2.869623] Key type encrypted registered [ 2.871525] ima: No TPM chip found, activating TPM-bypass! [ 2.873773] ima: Allocated hash algorithm: sha1 [ 2.875484] ima: No architecture policies found [ 2.877190] evm: Initialising EVM extended attributes: [ 2.878948] evm: security.selinux [ 2.880221] evm: security.ima [ 2.881457] evm: security.capability [ 2.882605] evm: HMAC attrs: 0x1 [ 2.884562] rtc_cmos 00:05: setting system clock to 2026-01-02 02:34:00 UTC (1767321240) [ 2.890442] debug: unmapping init [mem 0xffffffff8de03000-0xffffffff8dffffff] [ 2.893178] debug: unmapping init [mem 0xffffffff8cb82000-0xffffffff8ce58fff] [ 2.904112] Write protecting the kernel read-only data: 28672k [ 2.907408] debug: unmapping init [mem 0xffffffff8b203000-0xffffffff8b3fffff] [ 2.909740] debug: unmapping init [mem 0xffffffff8bb14000-0xffffffff8bbfffff] [ 2.935873] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 2.941841] systemd[1]: Detected virtualization kvm. [ 2.943317] systemd[1]: Detected architecture x86-64. [ 2.944639] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 2.966325] systemd[1]: No hostname configured. [ 2.967859] systemd[1]: Set hostname to . [ 2.969218] random: systemd: uninitialized urandom read (16 bytes read) [ 2.970883] systemd[1]: Initializing machine ID from random generator. [ 3.079685] random: systemd: uninitialized urandom read (16 bytes read) [ 3.082389] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.086806] random: systemd: uninitialized urandom read (16 bytes read) [ 3.089520] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.093331] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.693531] device-mapper: uevent: version 1.0.3 [ 3.695890] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System.[ 4.153504] random: fast init done [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.306715] virtio_net virtio0 ens2: renamed from eth0 [ 4.335245] scsi host0: ata_piix [ 4.383702] scsi host1: ata_piix [ 4.385274] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.387800] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.616050] dracut-initqueue[596]: RTNETLINK answers: File exists [ 9.386308] random: crng init done [ 9.387371] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 9.608503] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.666538] printk: systemd: 26 output lines suppressed due to ratelimiting [ 10.906734] SELinux: Disabled at runtime. [ 10.965729] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 10.974306] systemd[1]: Detected virtualization kvm. [ 10.976411] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.387768] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.390535] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.394165] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.398163] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.402804] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.408954] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.416903] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ 11.452314] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target rpc_pipefs.target. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 11.832846] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.077377] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.096494] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.214656] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.232101] EDAC sbridge: Ver: 1.1.2 [ 13.120233] Key type dns_resolver registered [ 13.411769] NFS: Registering the id_resolver key type [ 13.413151] Key type id_resolver registered [ 13.414441] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg356-client login: [ 42.161922] libcfs: loading out-of-tree module taints kernel. [ 42.215940] Key type ._llcrypt registered [ 42.219661] Key type .llcrypt registered [ 42.625824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 42.633675] alg: No test for adler32 (adler32-zlib) [ 43.683432] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 44.065854] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 45.703306] Key type lgssc registered [ 46.449078] Lustre: Echo OBD driver; http://www.lustre.org/ [ 85.951059] hrtimer: interrupt took 6840428 ns [ 261.932959] Lustre: Mounted lustre-client [ 267.273577] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 287.718436] Lustre: lustre-OST0000-osc-ffff9827d9cfd000: disconnect after 23s idle [ 289.470462] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing check_logdir /tmp/testlogs/ [ 294.470820] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing yml_node [ 301.822386] Lustre: DEBUG MARKER: Client: 2.17.0.RC4 [ 304.117525] Lustre: DEBUG MARKER: MDS: 2.17.0.RC4 [ 306.727678] Lustre: DEBUG MARKER: OSS: 2.17.0.RC4 [ 308.237990] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Jan 1 21:39:04 EST 2026 [ 324.296758] LustreError: 9708:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9827d9cfd000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 324.306555] LustreError: 9708:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 324.322298] Lustre: Unmounted lustre-client [ 349.493273] Key type lgssc unregistered [ 349.782595] LNet: 10241:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 349.791966] LNetError: 10241:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 349.826784] LNet: Removed LNI 192.168.203.56@tcp [ 350.566262] Key type .llcrypt unregistered [ 350.568664] Key type ._llcrypt unregistered [ 383.333835] Key type ._llcrypt registered [ 383.335287] Key type .llcrypt registered [ 383.642404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 383.675344] alg: No test for adler32 (adler32-zlib) [ 384.691588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 384.768900] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 384.782613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 388.459147] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 388.856012] LNet: 11203:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 388.873168] LNetError: 11203:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 388.897413] LNet: Removed LNI 192.168.203.56@tcp [ 389.578463] Key type .llcrypt unregistered [ 389.583145] Key type ._llcrypt unregistered [ 395.848661] Key type ._llcrypt registered [ 395.851180] Key type .llcrypt registered [ 397.215371] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 21:40:33 (1767321633) [ 397.652877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 397.667876] alg: No test for adler32 (adler32-zlib) [ 398.546789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.597798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 398.611735] LNet: 11611:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 398.625464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.655513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 398.666978] LNet: 11618:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 405.013276] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 21:40:41 (1767321641) [ 405.271879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 405.301334] LNet: 12196:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 405.326713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 405.399987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 412.033655] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 21:40:48 (1767321648) [ 412.179324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 412.210971] LNet: 12779:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 412.246663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 412.357326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 418.794352] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 21:40:55 (1767321655) [ 418.927770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 418.961570] LNet: 13360:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 418.997079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 419.074704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 425.419308] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 21:41:01 (1767321661) [ 425.611334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 425.626753] LNet: 13951:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 425.644514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.717801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 426.075392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 432.692545] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 21:41:09 (1767321669) [ 432.935286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.980319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 433.065422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 433.536964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 434.205255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 440.444961] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 21:41:16 (1767321676) [ 440.670643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 440.684802] LNet: 15137:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 440.693028] LNet: 15137:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 440.723606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 440.832281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 446.829486] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 21:41:23 (1767321683) [ 447.093283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 447.140372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 447.218255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 447.262945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 447.670606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 447.700458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 448.171200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 448.200569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 448.539033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 448.584187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 448.934349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 448.979969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 449.329809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 449.367062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 449.751347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 449.786791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 450.124572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 450.154652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 450.567882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 450.630530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 451.243841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 451.282891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 451.596521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 451.621991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 452.070441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 452.115931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 452.496783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 452.540713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 460.333227] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 21:41:36 (1767321696) [ 460.492857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.558089] LNet: 16404:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 460.575279] LNet: 16404:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 460.606786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.685663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 460.715878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 466.698362] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 21:41:43 (1767321703) [ 466.787522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 466.864294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 466.978751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 467.010893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 472.984896] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 21:41:49 (1767321709) [ 473.173900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 473.277240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 473.347440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 473.385617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 479.201366] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 21:41:55 (1767321715) [ 479.444802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 479.471846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.539336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 479.592954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 485.677354] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 21:42:02 (1767321722) [ 485.755102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.811670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.926372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 485.975546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 491.620499] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 21:42:08 (1767321728) [ 491.725542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.763547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.862888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 491.892942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 497.722115] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 21:42:14 (1767321734) [ 497.920117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 497.952798] LNet: 19923:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 497.957331] LNet: 19923:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 497.973776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 498.036637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 498.464116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 504.633852] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 21:42:21 (1767321741) [ 504.750046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 504.800330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.041763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 505.273438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 510.827558] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 21:42:27 (1767321747) [ 510.945187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 511.003289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 511.146743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 511.343500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 517.012091] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 21:42:33 (1767321753) [ 517.189435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 517.216200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 517.723769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 518.063189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 524.475700] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 21:42:40 (1767321760) [ 524.719446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 524.755598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 524.811975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 524.869420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 531.670943] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 21:42:48 (1767321768) [ 531.780508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 531.828040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 531.930447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 531.965399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 537.864734] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 21:42:54 (1767321774) [ 538.063882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 538.123237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 538.198578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 544.761712] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 21:43:00 (1767321780) [ 544.897846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 544.949470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 545.043128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 551.611806] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 21:43:07 (1767321787) [ 551.722938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 551.813139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 551.901099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 552.419398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 558.720344] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 21:43:15 (1767321795) [ 558.799920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 558.843598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.918492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 558.945563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 564.411861] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 21:43:20 (1767321800) [ 564.512464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 564.542609] LNet: 25943:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 564.545419] LNet: 25943:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 564.594308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 564.722384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 564.770484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 570.801912] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 21:43:27 (1767321807) [ 571.024190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 571.074021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 571.135900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 571.165921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 576.526852] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 21:43:33 (1767321813) [ 576.728541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 576.811843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 576.902431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 576.940809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 582.719429] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 21:43:39 (1767321819) [ 582.829511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 582.880282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 582.947570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 582.998560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 583.017060] LNetError: 27720:0:(peer.c:2225:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 583.049233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 583.068921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 588.392413] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 21:43:44 (1767321824) [ 588.461360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 588.501606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 588.598260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.VcRQKdQRTX [ 588.685309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.VcRQKdQRTX [ 588.736931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.VcRQKdQRTX [ 594.121745] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 21:43:50 (1767321830) [ 594.355042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 594.433307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.507386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 594.535499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 594.606273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 600.621580] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 21:43:56 (1767321836) [ 600.857028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 600.914118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 601.014527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 604.120820] LNet: Message forwarding will be enabled in 2s [ 607.200037] LNet: Message forwarding enabled [ 607.699382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 607.725763] LNet: Message forwarding will stop in 2s to 6s [ 612.241572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 612.645957] LNet: Message forwarding disabled [ 614.043296] LNet: Message forwarding will be enabled in 2s [ 617.759122] LNet: Message forwarding enabled [ 619.309633] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 21:44:15 (1767321855) [ 619.439697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.293295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.387940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 625.566165] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 21:44:21 (1767321861) [ 625.633627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 625.692972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 625.775538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 625.844465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 625.876206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 625.925331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 625.986945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 626.045909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 626.061198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 626.090429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 626.119103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 626.149943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 626.202116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 626.241931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 626.280916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 626.312501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 626.345853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 626.371472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 626.392456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 626.431680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 626.457875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 626.473699] LNetError: 30789:0:(router.c:629:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 631.354159] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 21:44:27 (1767321867) [ 631.487639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.519661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.607717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 636.706127] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 21:44:33 (1767321873) [ 638.313876] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg356-server [ 639.697156] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 21:44:36 (1767321876) [ 639.740489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 639.777474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.818379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 639.844231] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 641.073420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[57-61]@tcp [ 641.089230] LNet: 32504:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 641.150508] LNetError: 32291:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.57@tcp is being used as a gateway but routing feature is not turned on [ 641.170847] LNetError: 32291:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.57@tcp has gone from up to down [ 642.167214] LNetError: 32291:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.57@tcp is being used as a gateway but routing feature is not turned on [ 642.181688] LNetError: 32291:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 643.177934] LNetError: 32291:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.57@tcp is being used as a gateway but routing feature is not turned on [ 643.190558] LNetError: 32291:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 644.256627] LNetError: 32299:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.61@tcp has gone from up to down [ 644.270598] LNetError: 32299:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 645.227741] LNetError: 32291:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.58@tcp is being used as a gateway but routing feature is not turned on [ 645.241094] LNetError: 32291:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 647.062760] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 21:44:43 (1767321883) [ 648.696926] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg356-server [ 649.316969] LNetError: 32291:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.59@tcp is being used as a gateway but routing feature is not turned on [ 649.340691] LNetError: 32291:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 15 previous similar messages [ 650.118450] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 21:44:46 (1767321886) [ 650.184723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 650.335150] LNetError: 32290:0:(router.c:1267:lnet_check_routers()) Failed to discover router 192.168.203.57@tcp [ 651.235310] LNetError: 33434:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 651.252126] LNet: Removed LNI 192.168.203.56@tcp [ 651.315980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 651.391106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 651.422944] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 652.043943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[57-62/2]@tcp [ 652.068775] LNet: 33556:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 652.075184] LNet: 33556:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 652.106171] LNetError: 33443:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.59@tcp has gone from up to down [ 652.125516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[57-62/2]@tcp [ 657.571466] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 21:44:53 (1767321893) [ 657.672321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 657.700355] LNetError: 34145:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 657.732484] LNet: Removed LNI 192.168.203.56@tcp [ 657.755486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 657.847959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 657.912357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 657.944557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 658.025808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 658.072173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 658.102741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 658.137564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 658.206562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 658.248841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 658.311302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 658.374402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 658.399298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 658.437812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 658.511543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 658.538590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 658.581534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 658.633777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 658.666936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 658.715270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 658.783251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 658.822514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 658.888601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 658.968425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 659.016556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.064393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 659.147227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 659.193717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.280477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 659.377520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 659.396116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.437850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 659.480856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 659.510696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.556744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 664.319805] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 21:45:00 (1767321900) [ 665.616357] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg356-server [ 666.892745] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 21:45:03 (1767321903) [ 668.529289] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg356-server [ 669.857199] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 21:45:06 (1767321906) [ 670.384795] Key type .llcrypt unregistered [ 670.387207] Key type ._llcrypt unregistered [ 675.646225] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 675.866380] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 675.872606] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 675.874974] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 678.316289] Key type ._llcrypt registered [ 678.318655] Key type .llcrypt registered [ 678.477528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 678.501525] alg: No test for adler32 (adler32-zlib) [ 679.497961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 679.547833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 679.572447] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 680.250921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 680.284642] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 681.719373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 681.740847] LNet: Removed LNI 10.1.2.3@tcp [ 681.845399] LNet: 36263:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 681.848364] LNetError: 36263:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 681.866904] LNet: Removed LNI 192.168.203.56@tcp [ 682.283953] Key type .llcrypt unregistered [ 682.285334] Key type ._llcrypt unregistered [ 692.626662] Key type ._llcrypt registered [ 692.630350] Key type .llcrypt registered [ 693.594795] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 21:45:30 (1767321930) [ 694.341480] Key type .llcrypt unregistered [ 694.345856] Key type ._llcrypt unregistered [ 700.349568] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 700.493827] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 700.503567] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 700.514180] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 703.180291] Key type ._llcrypt registered [ 703.182795] Key type .llcrypt registered [ 703.395427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 703.419093] alg: No test for adler32 (adler32-zlib) [ 704.339711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 704.404920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 704.434404] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 705.428886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 705.452153] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 707.633717] LNet: 37750:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 707.637309] LNetError: 37750:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 707.645228] LNet: Removed LNI 192.168.203.56@tcp [ 707.652369] LNet: Removed LNI 10.1.2.3@tcp [ 708.258199] Key type .llcrypt unregistered [ 708.265771] Key type ._llcrypt unregistered [ 717.880052] Key type ._llcrypt registered [ 717.881635] Key type .llcrypt registered [ 719.026545] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 21:45:55 (1767321955) [ 720.826385] Key type .llcrypt unregistered [ 720.828792] Key type ._llcrypt unregistered [ 726.200485] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 726.334357] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 726.341345] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 726.344147] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 728.877651] Key type ._llcrypt registered [ 728.879606] Key type .llcrypt registered [ 729.066490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 729.086866] alg: No test for adler32 (adler32-zlib) [ 729.985133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.041862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 730.066152] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 731.904852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 731.929415] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 733.777933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 733.794562] LNet: Removed LNI 10.1.2.3@tcp [ 733.818282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 733.857822] LNet: Removed LNI 10.1.2.31@tcp [ 733.880721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 733.908722] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 734.587467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 734.610447] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 735.425807] LNet: 39659:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 735.434524] LNetError: 39659:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 735.438964] LNet: Removed LNI 10.1.2.3@tcp [ 735.450828] LNet: Removed LNI 10.1.2.31@tcp [ 736.000483] Key type .llcrypt unregistered [ 736.003903] Key type ._llcrypt unregistered [ 746.340855] Key type ._llcrypt registered [ 746.347216] Key type .llcrypt registered [ 747.573861] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 21:46:23 (1767321983) [ 748.268919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 748.288636] alg: No test for adler32 (adler32-zlib) [ 749.297204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 749.374904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 749.397978] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 750.517086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 750.536642] LNet: 40608:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 750.541150] LNetError: 40608:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 750.552891] LNet: Removed LNI 192.168.203.56@tcp [ 750.566497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 750.633459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.56@tcp [ 750.672172] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 751.620675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.56@tcp [ 751.676245] LNet: Removed LNI 192.168.203.56@tcp [ 751.759200] LNet: 40743:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 752.337253] Key type .llcrypt unregistered [ 752.339502] Key type ._llcrypt unregistered [ 765.538890] Key type ._llcrypt registered [ 765.545739] Key type .llcrypt registered [ 766.928427] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 21:46:42 (1767322002) [ 769.220174] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg356-server [ 771.236592] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 21:46:47 (1767322007) [ 771.688458] Key type .llcrypt unregistered [ 771.691574] Key type ._llcrypt unregistered [ 777.432233] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 777.542143] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 777.551399] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 777.584216] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 780.179048] Key type ._llcrypt registered [ 780.181721] Key type .llcrypt registered [ 780.413583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 780.430747] alg: No test for adler32 (adler32-zlib) [ 781.358694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.463882] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 782.280916] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 784.275817] LNet: 42575:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 784.284253] LNetError: 42575:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 784.288553] LNet: Removed LNI 192.168.203.56@tcp1 [ 784.295877] LNet: Removed LNI 10.1.2.3@tcp2 [ 784.836388] Key type .llcrypt unregistered [ 784.838379] Key type ._llcrypt unregistered [ 795.552557] Key type ._llcrypt registered [ 795.554152] Key type .llcrypt registered [ 796.650066] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 21:47:13 (1767322033) [ 796.913479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 796.922874] alg: No test for adler32 (adler32-zlib) [ 797.843262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 797.893357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 797.919949] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 798.666282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 798.678282] LNet: 43540:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 798.683579] LNetError: 43540:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 798.696818] LNet: Removed LNI 192.168.203.56@tcp [ 798.709428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 798.754524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 798.785063] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 805.151711] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 21:47:21 (1767322041) [ 805.247614] LNet: 44125:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 805.254636] LNetError: 44125:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 805.281885] LNet: Removed LNI 192.168.203.56@tcp [ 806.027394] Key type .llcrypt unregistered [ 806.032464] Key type ._llcrypt unregistered [ 811.363082] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 811.479202] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 811.481623] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 811.484594] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 814.133412] Key type ._llcrypt registered [ 814.139576] Key type .llcrypt registered [ 814.292700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 814.310459] alg: No test for adler32 (adler32-zlib) [ 815.210419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.288669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 815.331305] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 816.146955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 816.166794] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 817.912817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 817.935640] LNet: 44870:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 817.947572] LNetError: 44870:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 817.956299] LNet: Removed LNI 192.168.203.56@tcp [ 817.973676] LNet: Removed LNI 10.1.2.3@tcp [ 817.999419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 818.055958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 818.075500] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 818.079987] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 818.494492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 818.518228] LNet: 44898:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 818.527835] LNetError: 44898:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 818.533509] LNetError: 44898:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 818.541597] LNet: Removed LNI 192.168.203.56@tcp [ 818.556227] LNet: Removed LNI 10.1.2.3@tcp [ 818.567502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 818.623442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 818.646084] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 818.649850] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 819.107402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 819.122555] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 819.936657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 819.969473] LNet: 45060:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 819.977242] LNetError: 45060:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 819.983962] LNetError: 45060:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 819.994942] LNet: Removed LNI 192.168.203.56@tcp [ 819.998375] LNet: Removed LNI 10.1.2.3@tcp [ 820.014355] LNet: Removed LNI 192.168.203.56@tcp2 [ 820.053362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.108259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 820.124995] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 820.128876] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 820.132854] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 826.177393] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 21:47:42 (1767322062) [ 826.421167] LNet: 45657:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 826.425940] LNetError: 45657:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 826.430536] LNetError: 45657:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 826.434418] LNet: Removed LNI 192.168.203.56@tcp [ 826.437680] LNet: Removed LNI 10.1.2.3@tcp [ 826.451918] LNet: Removed LNI 192.168.203.56@tcp2 [ 826.922056] Key type .llcrypt unregistered [ 826.923714] Key type ._llcrypt unregistered [ 832.882591] Key type ._llcrypt registered [ 832.885510] Key type .llcrypt registered [ 833.157120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 833.169296] alg: No test for adler32 (adler32-zlib) [ 834.100211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 834.207272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 834.236193] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 835.043523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 835.080806] LNet: 46048:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 835.090197] LNetError: 46048:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 835.109279] LNet: Removed LNI 192.168.203.56@tcp [ 835.124115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 835.185702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 835.209867] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 835.942641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 836.008712] LNet: 46078:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 836.016971] LNetError: 46078:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 836.038514] LNet: Removed LNI 192.168.203.56@tcp [ 836.100839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 836.190850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 836.229045] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 836.853200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 836.928829] LNet: Removed LNI 192.168.203.56@tcp [ 836.946218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 837.043879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 837.079760] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 837.727526] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 837.962876] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 837.975582] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 837.996468] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 840.355328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 840.406176] LNet: 46181:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 840.417449] LNet: 46181:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 840.436924] LNetError: 46181:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 840.445575] LNetError: 46181:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 840.466742] LNet: Removed LNI 192.168.203.56@tcp [ 840.505618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 840.574957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 840.598494] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 847.166468] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 21:48:03 (1767322083) [ 847.459456] LNet: 46778:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 847.479391] LNetError: 46778:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 847.496421] LNet: Removed LNI 192.168.203.56@tcp [ 848.050782] Key type .llcrypt unregistered [ 848.053295] Key type ._llcrypt unregistered [ 854.709655] Key type ._llcrypt registered [ 854.719181] Key type .llcrypt registered [ 854.935192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 854.957242] alg: No test for adler32 (adler32-zlib) [ 855.973166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 856.063452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 856.110228] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 856.988684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 857.009548] LNet: 47173:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 857.016210] LNetError: 47173:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 857.030331] LNet: Removed LNI 192.168.203.56@tcp [ 857.045764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 857.113671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 857.138568] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 863.636928] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 21:48:19 (1767322099) [ 863.738557] LNet: 47756:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 863.762515] LNetError: 47756:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 863.800572] LNet: Removed LNI 192.168.203.56@tcp [ 864.384695] Key type .llcrypt unregistered [ 864.389131] Key type ._llcrypt unregistered [ 869.261236] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 869.334991] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 869.344771] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 869.352913] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 871.963468] Key type ._llcrypt registered [ 871.965821] Key type .llcrypt registered [ 872.136192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 872.160230] alg: No test for adler32 (adler32-zlib) [ 873.126066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 873.198091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 873.225933] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 873.994468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 874.010946] LNet: 48212:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 874.016840] LNetError: 48212:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 874.032160] LNet: Removed LNI 192.168.203.56@tcp [ 874.052663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 874.111806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 874.128083] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 880.865324] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 21:48:36 (1767322116) [ 880.997981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 881.035969] LNet: 48806:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.046549] LNetError: 48806:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 881.090650] LNet: Removed LNI 192.168.203.56@tcp [ 881.122240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.223512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 886.799283] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 21:48:43 (1767322123) [ 886.867676] LNet: 49384:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 887.541679] Key type .llcrypt unregistered [ 887.547710] Key type ._llcrypt unregistered [ 893.510792] Key type ._llcrypt registered [ 893.513221] Key type .llcrypt registered [ 893.731493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 893.749256] alg: No test for adler32 (adler32-zlib) [ 894.654150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 894.713225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 894.759559] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 895.465681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 895.490538] LNet: 49786:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 895.495854] LNetError: 49786:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 895.513852] LNet: Removed LNI 192.168.203.56@tcp [ 895.545296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.617233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 895.653204] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 902.626375] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 21:48:58 (1767322138) [ 902.769149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 902.820043] LNet: 50378:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 902.835614] LNetError: 50378:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 902.867202] LNet: Removed LNI 192.168.203.56@tcp [ 902.922174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 903.002781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 908.515803] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 21:49:04 (1767322144) [ 910.061231] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 911.511903] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 21:49:07 (1767322147) [ 912.641024] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 913.877691] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 21:49:10 (1767322150) [ 914.965842] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 916.328645] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 21:49:12 (1767322152) [ 917.876786] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 919.298521] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 21:49:15 (1767322155) [ 920.689112] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 922.023303] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 21:49:18 (1767322158) [ 922.115142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 922.167238] LNet: 52425:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 922.198521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 922.275680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 922.287111] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 922.991587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.56 [ 923.010988] LNet: 52566:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 923.024813] LNet: Removed LNI 192.168.203.56@tcp [ 928.367091] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 21:49:24 (1767322164) [ 928.499238] LNet: 53130:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 928.510183] LNetError: 53130:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 928.527813] LNet: Removed LNI 192.168.203.56@tcp [ 929.090414] Key type .llcrypt unregistered [ 929.092542] Key type ._llcrypt unregistered [ 934.934853] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 935.124521] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 935.128880] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 935.135161] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 937.831408] Key type ._llcrypt registered [ 937.835498] Key type .llcrypt registered [ 938.003499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 938.023682] alg: No test for adler32 (adler32-zlib) [ 939.338835] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 944.882427] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 21:49:41 (1767322181) [ 945.124684] LNet: 54038:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 945.132033] LNetError: 54038:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 945.150464] LNet: Removed LNI 10.1.2.3@tcp [ 945.742575] Key type .llcrypt unregistered [ 945.746268] Key type ._llcrypt unregistered [ 951.708378] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 951.931285] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 951.937240] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 951.942655] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 954.618916] Key type ._llcrypt registered [ 954.623137] Key type .llcrypt registered [ 954.754647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 954.767612] alg: No test for adler32 (adler32-zlib) [ 955.677725] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 960.726903] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 21:49:57 (1767322197) [ 960.938607] LNet: 54926:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 960.943657] LNetError: 54926:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 960.959303] LNet: Removed LNI 10.1.2.3@tcp [ 961.339284] Key type .llcrypt unregistered [ 961.343262] Key type ._llcrypt unregistered [ 966.617731] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 966.953822] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 966.961353] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 966.971397] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 968.555466] Key type ._llcrypt registered [ 968.558602] Key type .llcrypt registered [ 968.678934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 968.698981] alg: No test for adler32 (adler32-zlib) [ 969.620571] LNet: 55217:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 969.629893] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 975.320893] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 21:50:11 (1767322211) [ 975.579371] LNet: 55809:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 975.590496] LNetError: 55809:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 975.609841] LNet: Removed LNI 10.1.2.3@tcp [ 976.244456] Key type .llcrypt unregistered [ 976.246286] Key type ._llcrypt unregistered [ 982.113401] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 982.330775] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 982.335147] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 982.340492] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 983.946532] Key type ._llcrypt registered [ 983.948158] Key type .llcrypt registered [ 984.124914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 984.155638] alg: No test for adler32 (adler32-zlib) [ 985.040531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 985.089644] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 993.588847] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 21:50:30 (1767322230) [ 994.488525] LNet: 57027:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 994.496139] LNetError: 57027:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 994.512342] LNet: Removed LNI 10.1.2.3@tcp [ 995.263317] Key type .llcrypt unregistered [ 995.267443] Key type ._llcrypt unregistered [ 1002.928645] Key type ._llcrypt registered [ 1002.936562] Key type .llcrypt registered [ 1003.416397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1003.447608] alg: No test for adler32 (adler32-zlib) [ 1004.379633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1004.411112] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1011.405542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 1012.075297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_interrupt -r 1 [ 1012.109966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_interrupt -r 1 [ 1012.156698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1012.183842] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1012.448539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1012.475995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1012.503698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1012.550075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1012.574749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1012.614863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1012.651142] LNet: 1 local NIs in recovery (showing 1): 192.168.203.56@tcp [ 1012.663086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1012.692824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1012.716907] LNet: 57461:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1012.739920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1012.767864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1012.787193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1012.810126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1012.829777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1012.846277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1012.871931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1013.526349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_dropped -r 1 [ 1013.580964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_dropped -r 1 [ 1013.638392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1013.670783] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1013.924584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1013.958417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1013.999579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1014.041639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1014.072118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1014.120610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1014.159919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1014.180776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1014.194539] LNet: 57570:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1014.209707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1014.237635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1014.284809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1014.313686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1014.348180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1014.377186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1014.419343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1015.183608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_aborted -r 1 [ 1015.219989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_aborted -r 1 [ 1015.253568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1015.269455] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1015.627597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1015.693482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1015.728229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1015.788224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1015.848166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1015.895262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1015.928972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1015.973700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1016.002314] LNet: 57680:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1016.022849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1016.081646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1016.144498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1016.237592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1016.314370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1016.403775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1016.503715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1017.254602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_no_route -r 1 [ 1017.345640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_no_route -r 1 [ 1017.462893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1017.493416] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1017.728745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1017.771699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1017.810458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1017.835896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1017.883619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1017.947880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1017.987572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1018.033342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1018.067876] LNet: 57789:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1018.082470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1018.109633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1018.130526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1018.151028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1018.173477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1018.195555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1018.226949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1018.693492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_timeout -r 1 [ 1018.731559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_timeout -r 1 [ 1018.766884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1019.007854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1019.042976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1019.070245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1019.094535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1019.126841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1019.168425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1019.206979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1019.240963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1019.283989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1019.337858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1019.387888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1019.454637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1019.504945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1019.539372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1019.574599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1020.105069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_error -r 1 [ 1020.150685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_error -r 1 [ 1020.205327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1020.435147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1020.462366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1020.494189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1020.520236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1020.542495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1020.595868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1020.652994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1020.682095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1020.714172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1020.750509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1020.804422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1020.842603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1020.867574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1020.910211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1020.969881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1021.175925] LNet: 58048:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1021.180741] LNetError: 58048:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1021.196330] LNet: Removed LNI 192.168.203.56@tcp [ 1021.460679] Key type .llcrypt unregistered [ 1021.465953] Key type ._llcrypt unregistered [ 1037.204343] Key type ._llcrypt registered [ 1037.209516] Key type .llcrypt registered [ 1038.325741] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 21:51:14 (1767322274) [ 1039.178473] Key type .llcrypt unregistered [ 1039.180967] Key type ._llcrypt unregistered [ 1045.295649] Key type ._llcrypt registered [ 1045.300914] Key type .llcrypt registered [ 1045.482175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1045.496432] alg: No test for adler32 (adler32-zlib) [ 1046.374605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1046.409695] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1053.688656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 1062.615984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1062.646895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1062.667705] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 1063.812738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_interrupt -r 1 [ 1063.841602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e local_interrupt -r 1 [ 1063.871615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_interrupt -r 1 [ 1063.908912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e local_interrupt -r 1 [ 1063.939682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e local_interrupt -r 1 [ 1063.964684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e local_interrupt -r 1 [ 1063.988824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e local_interrupt -r 1 [ 1064.029952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e local_interrupt -r 1 [ 1064.092885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1064.111195] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1064.115408] LNet: 1 local NIs in recovery (showing 1): 192.168.203.56@tcp [ 1064.365986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1064.397825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1064.424684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1064.449877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1064.480072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1064.507161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1064.540242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1064.570993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1064.596719] LNet: 59510:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1064.619086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1064.646311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1064.680682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1064.718958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1064.750881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1064.783422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1064.819404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1065.360369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_dropped -r 1 [ 1065.392344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e local_dropped -r 1 [ 1065.431401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_dropped -r 1 [ 1065.471604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e local_dropped -r 1 [ 1065.503271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e local_dropped -r 1 [ 1065.531738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e local_dropped -r 1 [ 1065.566729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e local_dropped -r 1 [ 1065.592821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e local_dropped -r 1 [ 1065.616837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1065.627798] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1065.631206] LNet: Skipped 2 previous similar messages [ 1065.821475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1065.861150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1065.895779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1065.920959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1065.944745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1065.969309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1065.989570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1066.007972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1066.019374] LNet: 59632:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1066.033439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1066.067210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1066.107987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1066.134245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1066.160433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1066.196964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1066.233955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1066.639150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_aborted -r 1 [ 1066.678314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e local_aborted -r 1 [ 1066.720322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_aborted -r 1 [ 1066.747188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e local_aborted -r 1 [ 1066.779912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e local_aborted -r 1 [ 1066.805833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e local_aborted -r 1 [ 1066.828645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e local_aborted -r 1 [ 1066.854266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e local_aborted -r 1 [ 1066.879725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1066.894300] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1066.899973] LNet: Skipped 2 previous similar messages [ 1067.089980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1067.116298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1067.137854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1067.162577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1067.185416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1067.227527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1067.252515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1067.299348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1067.321809] LNet: 59753:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1067.343418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1067.378519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1067.404168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1067.422287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1067.435255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1067.450700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1067.470651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1067.875194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_no_route -r 1 [ 1067.921102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e local_no_route -r 1 [ 1067.963445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_no_route -r 1 [ 1067.996863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e local_no_route -r 1 [ 1068.026785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e local_no_route -r 1 [ 1068.063712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e local_no_route -r 1 [ 1068.090892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e local_no_route -r 1 [ 1068.123343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e local_no_route -r 1 [ 1068.159770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1068.319506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1068.338536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1068.360928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1068.380760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1068.403626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1068.416424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1068.447851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1068.472991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1068.493885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1068.519148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1068.549394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1068.575911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1068.602667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1068.625442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1068.650431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1069.068039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_timeout -r 1 [ 1069.112505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e local_timeout -r 1 [ 1069.194873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_timeout -r 1 [ 1069.230776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e local_timeout -r 1 [ 1069.264667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e local_timeout -r 1 [ 1069.301448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e local_timeout -r 1 [ 1069.336351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e local_timeout -r 1 [ 1069.385271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e local_timeout -r 1 [ 1069.419354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1069.440524] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1069.445237] LNet: Skipped 5 previous similar messages [ 1069.627209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1069.647429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1069.673631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1069.691858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1069.719906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1069.743834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1069.770057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1069.803174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1069.822739] LNet: 59995:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1069.828813] LNet: 59995:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 1069.853512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1069.879254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1069.910731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1069.936334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1069.955660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1069.982454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1070.011812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1070.398913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_error -r 1 [ 1070.432348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e local_error -r 1 [ 1070.456750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_error -r 1 [ 1070.504931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e local_error -r 1 [ 1070.548650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e local_error -r 1 [ 1070.592541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e local_error -r 1 [ 1070.634496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e local_error -r 1 [ 1070.663308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e local_error -r 1 [ 1070.688782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1070.844177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1070.881747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1070.906612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1070.945103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1070.988334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1071.008646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1071.045236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1071.070676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1071.093275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1071.117925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1071.140424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1071.162693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1071.186458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1071.210328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1071.234409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1074.937660] LNet: 60191:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1074.944560] LNetError: 60191:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1074.949930] LNet: Removed LNI 192.168.203.56@tcp [ 1074.961651] LNet: Removed LNI 192.168.203.56@tcp1 [ 1075.218565] Key type .llcrypt unregistered [ 1075.220446] Key type ._llcrypt unregistered [ 1088.530715] Key type ._llcrypt registered [ 1088.532980] Key type .llcrypt registered [ 1089.573714] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 21:52:06 (1767322326) [ 1090.396360] Key type .llcrypt unregistered [ 1090.400259] Key type ._llcrypt unregistered [ 1095.774921] Key type ._llcrypt registered [ 1095.777920] Key type .llcrypt registered [ 1095.985227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1096.000571] alg: No test for adler32 (adler32-zlib) [ 1096.898212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1096.919483] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1103.725972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 1104.340064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e remote_dropped -r 1 [ 1104.373224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e remote_dropped -r 1 [ 1104.422562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1104.449235] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1104.654858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1104.682159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1104.720754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1104.770783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1104.809697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1104.833380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1104.861406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1104.886380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1104.902584] LNet: 61434:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1104.920140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1104.961275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1104.996880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1105.025549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1105.049319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1105.072757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1105.098312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1105.575410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e remote_error -r 1 [ 1105.611895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e remote_error -r 1 [ 1105.646364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1105.673950] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1105.892415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1105.922598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1105.951205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1105.975181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1106.004774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1106.030134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1106.051935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1106.080890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1106.097197] LNet: 61543:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1106.120753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1106.151472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1106.177100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1106.203975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1106.233659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1106.258517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1106.284542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1106.712960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e remote_timeout -r 1 [ 1106.760599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e remote_timeout -r 1 [ 1106.799377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1106.824290] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1107.065627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1107.097742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1107.124359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1107.151408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1107.190952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1107.231365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1107.268477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1107.323353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1107.342347] LNet: 61653:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1107.361945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1107.395600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1107.432749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1107.465720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1107.501911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1107.538943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1107.593540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1107.896933] LNet: 61693:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1107.903920] LNetError: 61693:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1107.917744] LNet: Removed LNI 192.168.203.56@tcp [ 1108.254143] Key type .llcrypt unregistered [ 1108.255815] Key type ._llcrypt unregistered [ 1121.518931] Key type ._llcrypt registered [ 1121.521289] Key type .llcrypt registered [ 1122.626390] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 21:52:39 (1767322359) [ 1123.243613] Key type .llcrypt unregistered [ 1123.245277] Key type ._llcrypt unregistered [ 1128.909388] Key type ._llcrypt registered [ 1128.913908] Key type .llcrypt registered [ 1129.083494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1129.105393] alg: No test for adler32 (adler32-zlib) [ 1130.084516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1130.104922] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1137.906974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 1146.179557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1146.264794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1146.323915] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 1148.118649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e remote_dropped -r 1 [ 1148.155181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e remote_dropped -r 1 [ 1148.187480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e remote_dropped -r 1 [ 1148.220093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e remote_dropped -r 1 [ 1148.263957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e remote_dropped -r 1 [ 1148.319968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e remote_dropped -r 1 [ 1148.367192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e remote_dropped -r 1 [ 1148.443582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e remote_dropped -r 1 [ 1148.512342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1148.541628] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1148.557578] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.156@tcp [ 1148.832838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1148.864705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1148.901340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1148.929233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1148.959611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1148.994298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1149.028610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1149.068358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1149.082226] LNet: 63156:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1149.095581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1149.133772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1149.196186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1149.233930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1149.277277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1149.324575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1149.374602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1149.406255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1150.048274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e remote_error -r 1 [ 1150.085599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e remote_error -r 1 [ 1150.119355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e remote_error -r 1 [ 1150.159240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e remote_error -r 1 [ 1150.192367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e remote_error -r 1 [ 1150.231639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e remote_error -r 1 [ 1150.299249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e remote_error -r 1 [ 1150.358441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e remote_error -r 1 [ 1150.449197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1150.483660] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1150.489064] LNet: Skipped 2 previous similar messages [ 1150.728959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1150.779045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1150.808758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1150.853042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1150.881708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1150.942356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1150.998108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1151.040881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1151.067914] LNet: 63279:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1151.102875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1151.141810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1151.175433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1151.201968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1151.239134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1151.269881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1151.314230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1151.386391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1152.042905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e remote_timeout -r 1 [ 1152.094909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e remote_timeout -r 1 [ 1152.154856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e remote_timeout -r 1 [ 1152.182524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e remote_timeout -r 1 [ 1152.228484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e remote_timeout -r 1 [ 1152.267974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e remote_timeout -r 1 [ 1152.296691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e remote_timeout -r 1 [ 1152.347416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e remote_timeout -r 1 [ 1152.404420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1152.435333] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1152.683962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1152.712522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1152.743198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1152.779104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1152.809685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1152.846814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1152.877163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1152.909936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1152.929767] LNet: 63403:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1152.953992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1153.010955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1153.042703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1153.081362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1153.123602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1153.192941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1153.253317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1153.310268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1156.189663] LNet: 63479:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1156.197793] LNetError: 63479:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1156.208936] LNet: Removed LNI 192.168.203.56@tcp [ 1156.222373] LNet: Removed LNI 192.168.203.56@tcp1 [ 1156.501686] Key type .llcrypt unregistered [ 1156.506116] Key type ._llcrypt unregistered [ 1169.268607] Key type ._llcrypt registered [ 1169.272814] Key type .llcrypt registered [ 1170.530539] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 21:53:26 (1767322406) [ 1170.936437] Key type .llcrypt unregistered [ 1170.937803] Key type ._llcrypt unregistered [ 1175.582984] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1175.697597] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1175.707518] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1175.710021] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1178.230863] Key type ._llcrypt registered [ 1178.232505] Key type .llcrypt registered [ 1178.374582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1178.385841] alg: No test for adler32 (adler32-zlib) [ 1179.292193] LNet: 64550:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1179.306132] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1179.336639] LNet: 64564:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1179.341732] LNetError: 64564:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1179.353650] LNet: Removed LNI 192.168.203.56@tcp [ 1179.846631] Key type .llcrypt unregistered [ 1179.849290] Key type ._llcrypt unregistered [ 1184.861446] Key type ._llcrypt registered [ 1184.863729] Key type .llcrypt registered [ 1185.072834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1185.092348] alg: No test for adler32 (adler32-zlib) [ 1186.016427] LNet: 64818:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1186.026782] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1186.029704] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 1186.098699] LNet: 64832:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1186.106473] LNetError: 64832:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1186.114464] LNet: Removed LNI 192.168.203.56@tcp [ 1186.125420] LNet: Removed LNI 10.1.2.3@tcp1 [ 1186.823532] Key type .llcrypt unregistered [ 1186.825858] Key type ._llcrypt unregistered [ 1191.908507] Key type ._llcrypt registered [ 1191.911350] Key type .llcrypt registered [ 1192.045477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1192.057510] alg: No test for adler32 (adler32-zlib) [ 1192.942533] LNet: 65087:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1192.954165] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1192.957644] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1192.988909] LNet: 65101:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1192.993529] LNetError: 65101:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1192.996938] LNet: Removed LNI 192.168.203.56@tcp [ 1193.012304] LNet: Removed LNI 10.1.2.3@tcp [ 1193.538719] Key type .llcrypt unregistered [ 1193.542365] Key type ._llcrypt unregistered [ 1198.212581] Key type ._llcrypt registered [ 1198.214118] Key type .llcrypt registered [ 1198.332606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1198.343730] alg: No test for adler32 (adler32-zlib) [ 1199.224219] LNet: 65358:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1199.235669] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1199.239727] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1199.279479] LNet: 65372:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1199.282792] LNetError: 65372:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1199.287033] LNet: Removed LNI 192.168.203.56@tcp [ 1199.299943] LNet: Removed LNI 10.1.2.3@tcp [ 1199.762509] Key type .llcrypt unregistered [ 1199.764980] Key type ._llcrypt unregistered [ 1205.208092] Key type ._llcrypt registered [ 1205.211354] Key type .llcrypt registered [ 1205.447332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1205.470553] alg: No test for adler32 (adler32-zlib) [ 1206.374620] LNet: 65632:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1206.384878] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1206.388181] LNetError: 65632:0:(config.c:1560:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 1206.392986] LNetError: Error -100 starting up LNI tcp [ 1206.403451] LNet: Removed LNI 192.168.203.56@tcp [ 1207.055562] Key type .llcrypt unregistered [ 1207.058933] Key type ._llcrypt unregistered [ 1216.514848] Key type ._llcrypt registered [ 1216.520969] Key type .llcrypt registered [ 1217.749705] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 21:54:14 (1767322454) [ 1218.485319] Key type .llcrypt unregistered [ 1218.492773] Key type ._llcrypt unregistered [ 1224.753688] Key type ._llcrypt registered [ 1224.757361] Key type .llcrypt registered [ 1224.984594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1224.999701] alg: No test for adler32 (adler32-zlib) [ 1225.949247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1225.995870] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1233.506466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 1234.212582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e network_timeout -r 1 [ 1234.255095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e network_timeout -r 1 [ 1234.320574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 1234.344907] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1234.612650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1234.645280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1234.688592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1234.721711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1234.761634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1234.790803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1234.834821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1234.863854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1234.884756] LNet: 66883:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1234.899127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1234.936689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1234.973643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1235.004243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1235.042343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1235.079311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1235.412012] LNet: 66921:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1235.418314] LNetError: 66921:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1235.432755] LNet: Removed LNI 192.168.203.56@tcp [ 1235.738479] Key type .llcrypt unregistered [ 1235.748921] Key type ._llcrypt unregistered [ 1254.633789] Key type ._llcrypt registered [ 1254.642304] Key type .llcrypt registered [ 1254.886468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1254.910326] alg: No test for adler32 (adler32-zlib) [ 1255.809161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1255.854625] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1262.801933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 1272.022458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1272.079269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1272.126891] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 1273.446811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e network_timeout -r 1 [ 1273.482551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e network_timeout -r 1 [ 1273.546361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e network_timeout -r 1 [ 1273.610817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e network_timeout -r 1 [ 1273.648500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e network_timeout -r 1 [ 1273.681475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e network_timeout -r 1 [ 1273.728506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e network_timeout -r 1 [ 1273.793525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e network_timeout -r 1 [ 1273.849488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 1273.870419] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1274.091755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1274.122604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1274.146277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1274.187518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1274.210796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1274.245747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1274.300108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1274.336422] LNet: 1 local NIs in recovery (showing 1): 192.168.203.56@tcp [ 1274.348428] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.156@tcp [ 1274.351116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1274.385920] LNet: 67795:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1274.412309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1274.468924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1274.523128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1274.580730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1274.661557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1274.730577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1279.446847] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1279.455985] LNet: Skipped 2 previous similar messages [ 1279.808896] LNet: 67868:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1279.834808] LNetError: 67868:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1280.866759] LNet: Removed LNI 192.168.203.56@tcp [ 1280.871988] LNetError: 67868:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1280.885147] LNet: Removed LNI 192.168.203.56@tcp1 [ 1281.388742] Key type .llcrypt unregistered [ 1281.395848] Key type ._llcrypt unregistered [ 1295.256232] Key type ._llcrypt registered [ 1295.258327] Key type .llcrypt registered [ 1296.357466] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 21:55:32 (1767322532) [ 1296.637419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1296.649183] alg: No test for adler32 (adler32-zlib) [ 1297.555381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1297.589923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1297.616531] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1298.266707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1298.283227] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 1298.931853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 1298.983230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1299.029897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1299.066659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1299.107416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1299.147941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1299.173581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.56@tcp [ 1299.187872] LNetError: 68965:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.203.56@tcp to recovery [ 1299.615413] LNet: 1 local NIs in recovery (showing 1): 192.168.203.56@tcp [ 1299.624694] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 1301.665987] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 1302.588851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1305.759311] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 1306.049223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1306.073603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1306.126872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1306.141433] LNet: 69072:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1306.151806] LNetError: 69072:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1306.158929] LNet: Removed LNI 192.168.203.56@tcp [ 1306.176841] LNet: Removed LNI 192.168.203.56@tcp1 [ 1306.207247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1306.283823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1306.304693] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1306.890899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1306.914556] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 1307.616316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 1307.644340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1307.703261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1307.739070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1307.780370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1307.827759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1307.875454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1307.917140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.56@tcp [ 1307.945286] LNetError: 69344:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.203.56@tcp to recovery [ 1308.321047] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 1311.404426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1314.463893] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 1314.469351] LNet: Skipped 1 previous similar message [ 1319.281549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1319.313649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1319.354386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1324.768569] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 21:56:01 (1767322561) [ 1324.873726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1324.896505] LNet: 70057:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1324.904930] LNetError: 70057:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1324.912806] LNetError: 70057:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1324.918487] LNet: Removed LNI 192.168.203.56@tcp [ 1324.937241] LNet: Removed LNI 192.168.203.56@tcp1 [ 1324.976470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1325.050577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1325.077976] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1325.630498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1325.646707] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 1326.406864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 1326.469801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1326.510735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1326.542637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1326.595420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1326.642668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1326.696912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.56@tcp --health 0 [ 1326.829553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1327.071949] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.56@tcp [ 1327.075573] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 1327.080649] LNet: Skipped 1 previous similar message [ 1342.204795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1342.374916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1342.407526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1342.428693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.56@tcp --health 500 [ 1342.525970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1349.287494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1349.356316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1349.376724] LNet: 70552:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1349.381815] LNetError: 70552:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1349.386983] LNetError: 70552:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1349.394296] LNet: Removed LNI 192.168.203.56@tcp [ 1349.405755] LNet: Removed LNI 192.168.203.56@tcp1 [ 1349.429166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1349.481668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1349.501640] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1350.184504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1350.211900] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 1350.984212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 1351.027551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1351.084858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1351.112987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1351.148930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1351.173933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1351.206150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1351.238649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.56@tcp --health 0 [ 1352.543654] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 1352.552341] LNet: Skipped 3 previous similar messages [ 1353.701900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1363.745944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1363.770436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1363.804447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1368.888277] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 21:56:45 (1767322605) [ 1369.206542] LNet: 71609:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1369.211586] LNetError: 71609:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1369.215145] LNetError: 71609:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1369.222239] LNet: Removed LNI 192.168.203.56@tcp [ 1369.241501] LNet: Removed LNI 192.168.203.56@tcp1 [ 1369.864710] Key type .llcrypt unregistered [ 1369.866119] Key type ._llcrypt unregistered [ 1374.610641] Key type ._llcrypt registered [ 1374.612843] Key type .llcrypt registered [ 1374.762465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1374.776533] alg: No test for adler32 (adler32-zlib) [ 1375.670924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1375.695385] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1382.988778] Lustre: DEBUG MARKER: Initial discovery [ 1383.107771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.156@tcp [ 1384.814819] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1384.979265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1385.002208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.156@tcp [ 1385.025731] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1385.042691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1385.075373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.156@tcp -r 1 -m GET -l 3 [ 1385.122319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.156@tcp -r 1 -m GET -e local_error [ 1385.156431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.156@tcp -r 1 -m PUT -l 6 [ 1385.951162] LNet: 1 local NIs in recovery (showing 1): 192.168.203.56@tcp [ 1386.411074] Lustre: DEBUG MARKER: Force oleg356-client.virtnet to discover 192.168.203.156@tcp (in background) [ 1386.578230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.156@tcp [ 1387.663380] Lustre: DEBUG MARKER: Force oleg356-server to discover 192.168.203.56@tcp [ 1389.670527] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1390.209529] Lustre: DEBUG MARKER: Wait for 72365 [ 1391.568478] Lustre: DEBUG MARKER: Finished wait on 72365 [ 1391.680387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1391.747590] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1391.765116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1392.012881] LNet: 72814:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1392.018543] LNetError: 72814:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1393.062666] LNet: Removed LNI 192.168.203.56@tcp [ 1393.452717] Key type .llcrypt unregistered [ 1393.457606] Key type ._llcrypt unregistered [ 1408.271136] Key type ._llcrypt registered [ 1408.275588] Key type .llcrypt registered [ 1409.668092] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 21:57:25 (1767322645) [ 1410.227564] Key type .llcrypt unregistered [ 1410.228696] Key type ._llcrypt unregistered [ 1416.704539] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1416.817522] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1416.823456] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1416.827572] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1419.372135] Key type ._llcrypt registered [ 1419.374576] Key type .llcrypt registered [ 1419.504637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1419.517895] alg: No test for adler32 (adler32-zlib) [ 1420.455936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1420.495449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1420.515846] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1421.023790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1421.044318] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1428.163777] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 21:57:44 (1767322664) [ 1428.317788] LNet: 74887:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1428.322327] LNetError: 74887:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1428.325905] LNet: Removed LNI 192.168.203.56@tcp [ 1428.338554] LNet: Removed LNI 10.1.2.3@tcp [ 1428.846375] Key type .llcrypt unregistered [ 1428.847621] Key type ._llcrypt unregistered [ 1434.220194] Key type ._llcrypt registered [ 1434.222143] Key type .llcrypt registered [ 1434.408329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1434.423721] alg: No test for adler32 (adler32-zlib) [ 1435.336959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1435.372813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1435.400603] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1436.223656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1436.237794] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1437.194453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.56@tcp [ 1438.239214] LNetError: Unexpected error -22 connecting to 192.168.203.56@tcp at host 192.168.203.56:988 [ 1442.814270] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 21:57:59 (1767322679) [ 1443.084590] LNet: 76016:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1443.089195] LNetError: 76016:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1444.127378] LNet: Removed LNI 192.168.203.56@tcp [ 1444.134704] LNetError: 76016:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1444.151763] LNet: Removed LNI 10.1.2.3@tcp [ 1444.756719] Key type .llcrypt unregistered [ 1444.759398] Key type ._llcrypt unregistered [ 1449.898922] Key type ._llcrypt registered [ 1449.903706] Key type .llcrypt registered [ 1450.143253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1450.156553] alg: No test for adler32 (adler32-zlib) [ 1451.065755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1451.168679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1451.223083] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 1451.843079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1451.859329] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 1452.651556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.56@tcp1 --nid 192.168.203.56@tcp2 [ 1456.155496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.56@tcp1 192.168.203.56@tcp2 [ 1456.176403] LNetError: 76846:0:(lib-move.c:2264:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.56@tcp1 to 192.168.203.56@tcp2 - Route table may be misconfigured [ 1456.197340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.56@tcp2 192.168.203.56@tcp1 [ 1460.921259] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 21:58:17 (1767322697) [ 1461.071332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1461.106279] LNet: 77418:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1461.116379] LNetError: 77418:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1461.128345] LNet: Removed LNI 192.168.203.56@tcp1 [ 1462.194933] LNet: Removed LNI 192.168.203.56@tcp2 [ 1462.257599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1462.323308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1462.349924] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1463.049947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1463.066660] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 1463.797742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 1463.826951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.56@tcp 192.168.203.56@tcp [ 1463.857951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.56@tcp1 192.168.203.56@tcp1 [ 1463.904617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.56@tcp -d 192.168.203.56@tcp -e network_timeout [ 1463.931443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -e network_timeout [ 1463.967542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -e network_timeout [ 1463.994527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -e network_timeout [ 1464.028897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1464.044506] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 1464.062662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1464.153606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1464.241721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1464.351239] LNet: 1 local NIs in recovery (showing 1): 192.168.203.56@tcp [ 1469.288492] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 21:58:25 (1767322705) [ 1469.458218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1469.497712] LNet: 78280:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1469.507407] LNetError: 78280:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1469.511966] LNetError: 78280:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1469.517578] LNet: Removed LNI 192.168.203.56@tcp [ 1470.566110] LNet: Removed LNI 192.168.203.56@tcp1 [ 1470.592420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1470.680393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1470.907944] LNet: 78316:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1471.293444] Key type .llcrypt unregistered [ 1471.299105] Key type ._llcrypt unregistered [ 1479.892618] Key type ._llcrypt registered [ 1479.898178] Key type .llcrypt registered [ 1480.806658] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 21:58:37 (1767322717) [ 1481.111661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1481.120197] alg: No test for adler32 (adler32-zlib) [ 1482.046094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1483.142566] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1484.324315] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 21:58:40 (1767322720) [ 1484.436738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1484.456878] LNet: 79408:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1484.492096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1484.527840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1484.542710] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1485.111713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1485.129485] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 1485.693379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1485.723749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp1 [ 1485.753221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp1 [ 1491.026963] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 21:58:47 (1767322727) [ 1492.267681] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg356-server [ 1493.800538] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 21:58:50 (1767322730) [ 1495.094973] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg356-server [ 1496.558523] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 21:58:53 (1767322733) [ 1498.325445] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg356-server [ 1499.712909] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 21:58:56 (1767322736) [ 1501.041405] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg356-server [ 1502.502454] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 21:58:58 (1767322738) [ 1503.951789] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg356-server [ 1505.140920] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 21:59:01 (1767322741) [ 1506.505643] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg356-server [ 1507.865856] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 21:59:04 (1767322744) [ 1509.537666] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg356-server [ 1510.755329] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 21:59:07 (1767322747) [ 1512.017838] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg356-server [ 1513.125765] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 21:59:09 (1767322749) [ 1514.429435] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg356-server [ 1515.576761] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 21:59:12 (1767322752) [ 1515.743143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1515.770612] LNet: 83286:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1515.774686] LNetError: 83286:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1515.781351] LNet: Removed LNI 192.168.203.56@tcp [ 1516.839603] LNet: Removed LNI 192.168.203.56@tcp1 [ 1516.867112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1516.900613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1516.925400] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1517.370873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1517.423912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1517.564546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1517.581785] LNet: 83434:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1517.586371] LNetError: 83434:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1517.588822] LNetError: 83434:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1517.600625] LNet: Removed LNI 192.168.203.56@tcp [ 1517.612886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1517.650645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1517.667904] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1518.193460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1518.345430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1519.397542] LNet: Removed LNI 192.168.203.56@tcp [ 1519.435969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1519.499936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1519.536433] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1520.076127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1520.127895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1520.279592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1520.299294] LNet: 83726:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1520.303746] LNet: 83726:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1520.308655] LNetError: 83726:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1520.313902] LNetError: 83726:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1520.324935] LNet: Removed LNI 192.168.203.56@tcp [ 1520.352905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1520.385771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1520.416970] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1521.016317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1521.131493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1522.225645] LNet: Removed LNI 192.168.203.56@tcp [ 1522.243280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1522.312652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1522.347804] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1522.913876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1522.976095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1523.147289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1523.165278] LNetError: 84019:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1523.170865] LNetError: 84019:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1523.185350] LNet: Removed LNI 192.168.203.56@tcp [ 1523.198432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1523.240803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1523.271695] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1523.846919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1524.018587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1525.092844] LNet: Removed LNI 192.168.203.56@tcp [ 1525.109698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1525.179665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1525.200899] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1525.745358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1525.809657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1526.012547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1526.040206] LNet: 84311:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1526.044100] LNet: 84311:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1527.087936] LNet: Removed LNI 192.168.203.56@tcp [ 1527.114220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1527.191799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1527.237753] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1528.067207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1528.252383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1528.264295] LNetError: 84456:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1528.267744] LNetError: 84456:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1529.324714] LNet: Removed LNI 192.168.203.56@tcp [ 1529.348902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1529.410307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1529.433054] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1530.056425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1530.122733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1530.297987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1531.368919] LNet: Removed LNI 192.168.203.56@tcp [ 1531.382788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1531.422256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1531.441291] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1532.053430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1532.297638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1533.360432] LNet: Removed LNI 192.168.203.56@tcp [ 1533.379926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1533.411870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1533.457593] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1534.157749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1534.249693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1534.512373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1534.532530] LNet: 84896:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1534.545354] LNet: 84896:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1535.591375] LNet: Removed LNI 192.168.203.56@tcp [ 1535.611562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1535.697782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1535.732835] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1536.439979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1536.622214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1536.638670] LNetError: 85042:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1536.642298] LNetError: 85042:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1536.656564] LNet: Removed LNI 192.168.203.56@tcp [ 1536.681832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1536.741669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1536.776069] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1537.357886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1537.400335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1537.434769] LNet: Removed LNI 192.168.203.56@tcp [ 1537.461861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1537.532752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1537.555821] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1538.162270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1542.970238] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 21:59:39 (1767322779) [ 1543.054223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1543.088698] LNet: Removed LNI 192.168.203.56@tcp [ 1543.108580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1543.168542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1543.190196] LNet: Added LNI 192.168.203.56@tcp231 [8/256/0/180] [ 1544.065125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1544.098038] LNet: Removed LNI 192.168.203.56@tcp231 [ 1544.112586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1544.188419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1544.208994] LNet: Added LNI 192.168.203.56@tcp231 [8/256/0/180] [ 1544.213317] LNet: 86024:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1544.629301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1544.676898] LNet: Removed LNI 192.168.203.56@tcp231 [ 1544.694887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1544.720278] LNet: Added LNI 192.168.203.56@tcp231 [8/256/0/180] [ 1545.527711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1545.563814] LNet: Removed LNI 192.168.203.56@tcp231 [ 1545.577859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1545.639681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1545.664658] LNet: Added LNI 192.168.203.56@tcp231 [8/256/0/0] [ 1545.669461] LNet: 86176:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1545.926664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1545.951599] LNet: Removed LNI 192.168.203.56@tcp231 [ 1545.965941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1545.994182] LNet: Added LNI 192.168.203.56@tcp231 [8/256/0/0] [ 1546.827973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1546.851165] LNet: Removed LNI 192.168.203.56@tcp231 [ 1546.872705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1546.943575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1546.972379] LNet: Added LNI 192.168.203.56@tcp231 [8/256/0/60] [ 1546.977267] LNet: 86327:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1547.313904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1547.341541] LNet: Removed LNI 192.168.203.56@tcp231 [ 1547.354551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1547.369135] LNet: Added LNI 192.168.203.56@tcp231 [8/256/0/60] [ 1548.139640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1548.173947] LNet: Removed LNI 192.168.203.56@tcp231 [ 1548.197183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1548.253819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1548.268563] LNet: Added LNI 192.168.203.56@tcp231 [8/256/0/180] [ 1548.528118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1548.560236] LNet: Removed LNI 192.168.203.56@tcp231 [ 1548.578405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1548.600959] LNet: Added LNI 192.168.203.56@tcp231 [8/256/0/180] [ 1553.552825] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 21:59:50 (1767322790) [ 1553.733793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1553.765980] LNet: 87189:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1553.773901] LNet: 87189:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 1553.778903] LNetError: 87189:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1553.792957] LNetError: 87189:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1553.814937] LNet: Removed LNI 192.168.203.56@tcp231 [ 1553.843652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1553.904667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1553.925196] LNet: Added LNI 192.168.203.56@tcp232 [8/256/0/180] [ 1554.404547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1554.434274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1554.457785] LNet: Removed LNI 192.168.203.56@tcp232 [ 1554.467674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1554.483429] LNet: Added LNI 192.168.203.56@tcp232 [8/256/0/180] [ 1554.978482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1555.016898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1555.048303] LNet: Removed LNI 192.168.203.56@tcp232 [ 1555.064606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1555.092407] LNet: Added LNI 192.168.203.56@tcp232 [8/256/0/180] [ 1555.669126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1555.712955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1555.736327] LNet: Removed LNI 192.168.203.56@tcp232 [ 1555.752437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1555.772871] LNet: Added LNI 192.168.203.56@tcp232 [8/256/0/180] [ 1556.452883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1556.481435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1556.514526] LNet: Removed LNI 192.168.203.56@tcp232 [ 1556.528916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1561.649587] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 21:59:58 (1767322798) [ 1562.406395] Key type .llcrypt unregistered [ 1562.411325] Key type ._llcrypt unregistered [ 1565.899334] Key type ._llcrypt registered [ 1565.903415] Key type .llcrypt registered [ 1566.104436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1566.114663] alg: No test for adler32 (adler32-zlib) [ 1567.000635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1567.024228] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1571.730782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 1577.510462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1577.549950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1577.566733] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 1578.547860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_interrupt -r 2 [ 1578.567299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e local_interrupt -r 2 [ 1578.604931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_interrupt -r 2 [ 1578.627988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e local_interrupt -r 2 [ 1578.654806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e local_interrupt -r 2 [ 1578.676693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e local_interrupt -r 2 [ 1578.697765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e local_interrupt -r 2 [ 1578.718511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e local_interrupt -r 2 [ 1578.742497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.56@tcp 192.168.203.156@tcp [ 1578.754567] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1578.758386] LNet: 1 local NIs in recovery (showing 1): 192.168.203.56@tcp [ 1578.876933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1578.895964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1578.910162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1578.929489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1578.947565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1578.964993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1578.981391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1578.999069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1579.009225] LNet: 88974:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1579.022266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1579.040271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1579.071509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1579.088716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1579.105408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1579.123983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1579.376278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_dropped -r 2 [ 1579.400155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e local_dropped -r 2 [ 1579.419658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_dropped -r 2 [ 1579.438316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e local_dropped -r 2 [ 1579.457919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e local_dropped -r 2 [ 1579.477079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e local_dropped -r 2 [ 1579.495314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e local_dropped -r 2 [ 1579.515943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e local_dropped -r 2 [ 1579.537296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.56@tcp 192.168.203.156@tcp [ 1579.548940] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1579.686332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1579.709488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1579.727565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1579.747808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1579.764438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1579.785318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1579.808069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1579.825331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1579.835906] LNet: 89094:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1579.851462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1579.874423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1579.898221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1579.918101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1579.933439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1579.953664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1580.183295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_aborted -r 2 [ 1580.202076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e local_aborted -r 2 [ 1580.219825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_aborted -r 2 [ 1580.238298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e local_aborted -r 2 [ 1580.254711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e local_aborted -r 2 [ 1580.273785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e local_aborted -r 2 [ 1580.294939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e local_aborted -r 2 [ 1580.317145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e local_aborted -r 2 [ 1580.340326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.56@tcp 192.168.203.156@tcp [ 1580.488157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1580.508093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1580.530521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1580.562744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1580.594661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1580.611088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1580.631263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1580.659898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1580.683471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1580.712357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1580.746823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1580.772976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1580.803266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1580.832654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1581.164907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_no_route -r 2 [ 1581.190760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e local_no_route -r 2 [ 1581.228717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_no_route -r 2 [ 1581.269199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e local_no_route -r 2 [ 1581.294523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e local_no_route -r 2 [ 1581.311647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e local_no_route -r 2 [ 1581.335645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e local_no_route -r 2 [ 1581.358503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e local_no_route -r 2 [ 1581.377452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.56@tcp 192.168.203.156@tcp [ 1581.387717] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1581.394364] LNet: Skipped 2 previous similar messages [ 1581.512029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1581.529635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1581.546395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1581.563095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1581.584703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1581.603436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1581.634785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1581.657408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1581.672071] LNet: 89334:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1581.675650] LNet: 89334:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 1581.685261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1581.703400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1581.721042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1581.736577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1581.755260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1581.771531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1582.045794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_timeout -r 2 [ 1582.073811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e local_timeout -r 2 [ 1582.091521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_timeout -r 2 [ 1582.113342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e local_timeout -r 2 [ 1582.136977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e local_timeout -r 2 [ 1582.159781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e local_timeout -r 2 [ 1582.180469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e local_timeout -r 2 [ 1582.209059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e local_timeout -r 2 [ 1582.233413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.56@tcp 192.168.203.156@tcp [ 1582.379480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1582.395795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1582.415594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1582.440100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1582.480598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1582.499242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1582.534553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1582.561523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1582.596758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1582.624359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1582.661428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1582.685434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1582.712904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1582.743303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1583.185374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e remote_dropped -r 2 [ 1583.212448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e remote_dropped -r 2 [ 1583.244514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e remote_dropped -r 2 [ 1583.269686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e remote_dropped -r 2 [ 1583.298392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e remote_dropped -r 2 [ 1583.324510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e remote_dropped -r 2 [ 1583.386679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e remote_dropped -r 2 [ 1583.415914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e remote_dropped -r 2 [ 1583.444869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.56@tcp 192.168.203.156@tcp [ 1583.464192] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1583.468868] LNet: Skipped 2 previous similar messages [ 1583.471298] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.156@tcp [ 1583.610319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1583.630221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1583.652945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1583.681715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1583.701957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1583.721198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1583.749379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1583.769723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1583.777185] LNet: 89574:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1583.779834] LNet: 89574:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 1583.789456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1583.807028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1583.836910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1583.856990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1583.875777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1583.900865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1586.766266] LNet: 89647:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1586.773628] LNetError: 89647:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1586.780853] LNet: Removed LNI 192.168.203.56@tcp [ 1586.792637] LNet: Removed LNI 192.168.203.56@tcp1 [ 1587.113736] Key type .llcrypt unregistered [ 1587.116416] Key type ._llcrypt unregistered [ 1602.514195] Key type ._llcrypt registered [ 1602.515670] Key type .llcrypt registered [ 1603.693924] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 22:00:40 (1767322840) [ 1604.030349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1604.039753] alg: No test for adler32 (adler32-zlib) [ 1604.918544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1604.993367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1605.019328] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1605.605224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1605.626145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.56@tcp [ 1605.646525] LNetError: 90623:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.203.56@tcp to recovery [ 1605.694929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.56@tcp [ 1605.749793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1610.402610] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 22:00:46 (1767322846) [ 1610.592532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1610.608416] LNet: 91212:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1610.614755] LNetError: 91212:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1610.634114] LNet: Removed LNI 192.168.203.56@tcp [ 1610.657439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1610.715833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1610.735417] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1611.172132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1611.187874] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 1611.808484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.56@tcp [ 1611.817908] LNetError: 91478:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.203.56@tcp to recovery [ 1612.768477] LNet: 1 local NIs in recovery (showing 1): 192.168.203.56@tcp [ 1612.779068] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 1612.989759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.56@tcp [ 1620.150426] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 22:00:56 (1767322856) [ 1621.576582] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg356-server [ 1623.088099] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 22:00:59 (1767322859) [ 1623.255417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1623.283721] LNet: 92433:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1623.293645] LNetError: 92433:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1623.300342] LNet: Removed LNI 192.168.203.56@tcp [ 1623.318499] LNet: Removed LNI 192.168.203.56@tcp1 [ 1623.341793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1623.378573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1623.416187] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1624.003624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1624.074802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1624.127452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1624.177140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1624.199814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1624.224778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1628.774978] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 22:01:05 (1767322865) [ 1628.936761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1628.955360] LNet: 93171:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1628.960882] LNetError: 93171:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1628.965973] LNetError: 93171:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1628.979390] LNet: Removed LNI 192.168.203.56@tcp [ 1628.999430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1629.052814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1629.071788] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1633.458127] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 22:01:10 (1767322870) [ 1633.517297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1633.537854] LNet: 93868:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1633.543549] LNetError: 93868:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1633.570424] LNet: Removed LNI 192.168.203.56@tcp [ 1633.592170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1633.647402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1633.664350] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1637.993707] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 22:01:14 (1767322874) [ 1638.971053] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1640.053702] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 22:01:16 (1767322876) [ 1640.390706] LNet: 94908:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1640.393346] LNetError: 94908:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1640.405794] LNet: Removed LNI 192.168.203.56@tcp [ 1640.881814] Key type .llcrypt unregistered [ 1640.886963] Key type ._llcrypt unregistered [ 1644.764513] Key type ._llcrypt registered [ 1644.766454] Key type .llcrypt registered [ 1644.893894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1644.904539] alg: No test for adler32 (adler32-zlib) [ 1645.791996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1645.807870] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1650.433618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 1653.084687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.156@tcp [ 1653.294579] LNet: 95318:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1653.298477] LNetError: 95318:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1653.321744] LNet: Removed LNI 192.168.203.56@tcp [ 1653.596719] Key type .llcrypt unregistered [ 1653.601306] Key type ._llcrypt unregistered [ 1659.596498] Key type ._llcrypt registered [ 1659.598612] Key type .llcrypt registered [ 1660.432572] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1661.430421] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1662.402949] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 22:01:39 (1767322899) [ 1662.654664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1662.663541] alg: No test for adler32 (adler32-zlib) [ 1663.533133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1663.598796] LNet: 96400:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1663.952406] Key type .llcrypt unregistered [ 1663.954442] Key type ._llcrypt unregistered [ 1667.760276] Key type ._llcrypt registered [ 1667.763158] Key type .llcrypt registered [ 1667.877313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1667.891119] alg: No test for adler32 (adler32-zlib) [ 1668.737620] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1668.740270] LNet: 96652:0:(config.c:1226:lnet_parse_routes()) Kernel parsing of LNet routes is deprecated. Consider converting to LNet YAML configuration. [ 1668.746737] LNet: 96652:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1668.803619] LNetError: 96664:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.59@tcp is being used as a gateway but routing feature is not turned on [ 1668.820556] LNetError: 96664:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.203.59@tcp has gone from up to down [ 1669.127945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1687.583284] LNetError: 96664:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.203.61@tcp has gone from up to down [ 1687.592365] LNetError: 96664:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 1687.599611] LNet: 96710:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1687.607264] LNetError: 96710:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1687.634747] LNet: Removed LNI 192.168.203.56@tcp [ 1687.654680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1687.689911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1687.702650] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1687.706628] LNet: 96719:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1687.711882] LNet: 96719:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 1687.725973] LNetError: 96719:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.203.57@tcp is a gateway. Can not delete it [ 1687.739292] LNetError: 96719:0:(peer.c:2217:lnet_del_peer_ni()) 192.168.203.57@tcp is a router. Can not be deleted [ 1687.754384] LNet: 96719:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1687.757258] LNetError: 96716:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.60@tcp is being used as a gateway but routing feature is not turned on [ 1687.768354] LNetError: 96716:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 2 previous similar messages [ 1690.783514] LNetError: 96730:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.203.58@tcp has gone from up to down [ 1690.802144] LNetError: 96730:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 1693.188915] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 22:02:09 (1767322929) [ 1694.583324] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg356-server [ 1695.815447] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 22:02:12 (1767322932) [ 1697.056917] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg356-server [ 1698.102765] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 22:02:14 (1767322934) [ 1699.337094] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 1701.049732] Key type lgssc registered [ 1701.745421] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1743.672333] Key type lgssc unregistered [ 1743.892612] LNet: 99605:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1743.896149] LNetError: 99605:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1743.910739] LNet: Removed LNI 192.168.203.56@tcp [ 1744.258554] Key type .llcrypt unregistered [ 1744.260769] Key type ._llcrypt unregistered [ 1753.249726] Key type ._llcrypt registered [ 1753.251869] Key type .llcrypt registered [ 1754.023091] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 22:03:10 (1767322990) [ 1754.297679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1754.306319] alg: No test for adler32 (adler32-zlib) [ 1755.172306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1755.217775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1755.248687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1755.276307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1755.302839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1759.172807] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 22:03:15 (1767322995) [ 1759.358430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1759.382166] LNet: 101007:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1759.401510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1759.451207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1759.468356] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1760.090034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1760.111992] LNet: 101151:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1760.118118] LNetError: 101151:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1760.133685] LNet: Removed LNI 192.168.203.56@tcp [ 1760.163466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1760.206876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1760.227651] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1760.499493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1760.529551] LNet: Removed LNI 192.168.203.56@tcp [ 1760.548619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1760.602454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1760.615875] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1764.423219] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 22:03:21 (1767323001) [ 1764.483100] LNet: 101760:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1764.487951] LNet: 101760:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1764.493040] LNetError: 101760:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1764.499093] LNetError: 101760:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1764.518140] LNet: Removed LNI 192.168.203.56@tcp [ 1764.918240] Key type .llcrypt unregistered [ 1764.920262] Key type ._llcrypt unregistered [ 1768.783551] Key type ._llcrypt registered [ 1768.785694] Key type .llcrypt registered [ 1768.927625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1768.937564] alg: No test for adler32 (adler32-zlib) [ 1769.789110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1769.836412] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1769.839822] LNet: *** cfs_fail_loc=e003, val=100*** [ 1770.403766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1770.421879] LNet: *** cfs_fail_loc=e003, val=100*** [ 1770.427060] LNet: Skipped 2 previous similar messages [ 1770.432398] Ping reply received with mapping: [ 1770.433557] found metadata reply NI[192.168.203.56@tcp] [ 1770.437128] 38 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1770.439790] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1770.441890] 5a 5a 5a 5a 5a 5a 5a 5a [ 1770.521870] LNet: 102163:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1770.525410] LNetError: 102163:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1770.533840] LNet: Removed LNI 192.168.203.56@tcp [ 1770.849985] Key type .llcrypt unregistered [ 1770.852360] Key type ._llcrypt unregistered [ 1778.824284] Key type ._llcrypt registered [ 1778.828234] Key type .llcrypt registered [ 1779.871340] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 22:03:36 (1767323016) [ 1780.145568] Key type .llcrypt unregistered [ 1780.148599] Key type ._llcrypt unregistered [ 1783.817838] Key type ._llcrypt registered [ 1783.819625] Key type .llcrypt registered [ 1783.919109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1783.927988] alg: No test for adler32 (adler32-zlib) [ 1784.933170] LNetError: 103197:0:(api-ni.c:2732:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1785.201428] Key type .llcrypt unregistered [ 1785.202449] Key type ._llcrypt unregistered [ 1788.653556] Key type ._llcrypt registered [ 1788.654745] Key type .llcrypt registered [ 1789.786837] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 22:03:46 (1767323026) [ 1790.229468] Key type .llcrypt unregistered [ 1790.232110] Key type ._llcrypt unregistered [ 1794.072155] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1794.074933] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1796.385615] Key type ._llcrypt registered [ 1796.387379] Key type .llcrypt registered [ 1796.518123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1796.531436] alg: No test for adler32 (adler32-zlib) [ 1797.388702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1797.434256] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1803.642922] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 22:04:00 (1767323040) [ 1803.750601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1803.776610] LNet: 104965:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1803.781169] LNetError: 104965:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1803.796303] LNet: Removed LNI 10.1.2.3@tcp [ 1803.814081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1804.343119] Key type .llcrypt unregistered [ 1804.348753] Key type ._llcrypt unregistered [ 1808.692137] Key type ._llcrypt registered [ 1808.693761] Key type .llcrypt registered [ 1808.777699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1808.786075] alg: No test for adler32 (adler32-zlib) [ 1809.654104] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1809.716333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1809.824436] LNet: 105254:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1809.832646] LNetError: 105254:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1809.844690] LNet: Removed LNI 192.168.203.56@tcp [ 1810.274437] Key type .llcrypt unregistered [ 1810.276127] Key type ._llcrypt unregistered [ 1816.736220] Key type ._llcrypt registered [ 1816.738205] Key type .llcrypt registered [ 1817.576711] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 22:04:14 (1767323054) [ 1818.013371] Key type .llcrypt unregistered [ 1818.014897] Key type ._llcrypt unregistered [ 1822.001595] Key type ._llcrypt registered [ 1822.004465] Key type .llcrypt registered [ 1822.125905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1822.136837] alg: No test for adler32 (adler32-zlib) [ 1827.102196] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 22:04:23 (1767323063) [ 1827.150525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1827.181662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1827.238539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1827.255198] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1827.696063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1827.720277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1827.746738] LNet: Removed LNI 192.168.203.56@tcp [ 1827.894927] LNet: 106988:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1828.138477] Key type .llcrypt unregistered [ 1828.140477] Key type ._llcrypt unregistered [ 1834.329354] Key type ._llcrypt registered [ 1834.331131] Key type .llcrypt registered [ 1835.030375] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 22:04:31 (1767323071) [ 1835.298900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1835.305593] alg: No test for adler32 (adler32-zlib) [ 1836.162767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1836.212947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1836.230637] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1836.676802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1836.701132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.56@tcp [ 1836.712488] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1836.714889] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1836.716307] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1836, 0, 0 [ 1836.718566] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1836.720765] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1836.723217] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1836.725189] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1836, 0, 0 [ 1836.729243] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1836.731089] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1836.733052] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1836.734732] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1836, 0, 1886 [ 1836.737850] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1836.740574] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1836.743654] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1836.746121] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1836, 0, 1886 [ 1836.750105] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1840.778215] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 22:04:37 (1767323077) [ 1841.144855] LNet: 108524:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1841.151458] LNetError: 108524:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1841.168986] LNet: Removed LNI 192.168.203.56@tcp [ 1841.690383] Key type .llcrypt unregistered [ 1841.693110] Key type ._llcrypt unregistered [ 1845.991341] Key type ._llcrypt registered [ 1845.994204] Key type .llcrypt registered [ 1846.126659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1846.136170] alg: No test for adler32 (adler32-zlib) [ 1847.005894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1847.034899] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1852.260777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 1858.463787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1858.512072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1858.544814] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 1859.206126] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1859.314371] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1859.327123] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1859.336478] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1861.609108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1861.632996] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1868.845289] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 22:05:05 (1767323105) [ 1869.016880] LNet: 110027:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1869.022738] LNetError: 110027:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1869.027840] LNet: Removed LNI 192.168.203.56@tcp [ 1870.047231] LNet: Removed LNI 192.168.203.56@tcp1 [ 1870.048955] LNetError: 110027:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1870.057569] LNetError: 110027:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1870.068503] LNet: Removed LNI 10.1.2.3@tcp99 [ 1870.483931] Key type .llcrypt unregistered [ 1870.486882] Key type ._llcrypt unregistered [ 1874.793629] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1874.799346] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1877.225329] Key type ._llcrypt registered [ 1877.227580] Key type .llcrypt registered [ 1877.414587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1877.431861] alg: No test for adler32 (adler32-zlib) [ 1878.314147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1878.359837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1878.384394] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1878.826527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1878.835541] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1880.223506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1880.245564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1880.274777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1880.297455] LNetError: 110768:0:(peer.c:2225:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1880.316646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1880.341223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 1880.599046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1880.665620] LNet: 110788:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1880.669544] LNetError: 110788:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1880.677579] LNet: Removed LNI 192.168.203.56@tcp [ 1880.698086] LNet: Removed LNI 10.1.2.3@tcp [ 1881.073528] Key type .llcrypt unregistered [ 1881.075772] Key type ._llcrypt unregistered [ 1888.918415] Key type ._llcrypt registered [ 1888.929322] Key type .llcrypt registered [ 1890.020324] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 22:05:26 (1767323126) [ 1890.364852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1890.373883] alg: No test for adler32 (adler32-zlib) [ 1891.306659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1891.350695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1891.378783] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1892.084980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1892.129367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg356-client@tcp [ 1897.518531] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 22:05:34 (1767323134) [ 1897.660454] LNet: 112317:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1897.666946] LNetError: 112317:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1897.685336] LNet: Removed LNI 192.168.203.56@tcp [ 1898.386445] Key type .llcrypt unregistered [ 1898.388205] Key type ._llcrypt unregistered [ 1904.764995] Key type ._llcrypt registered [ 1904.770402] Key type .llcrypt registered [ 1904.956964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1904.969653] alg: No test for adler32 (adler32-zlib) [ 1905.882903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1905.909037] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1905.944336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1911.461903] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 22:05:47 (1767323147) [ 1912.341390] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1913.342961] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 22:05:50 (1767323150) [ 1914.345682] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1915.460542] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 22:05:52 (1767323152) [ 1915.531261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1915.550141] LNet: 113730:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1915.560496] LNetError: 113730:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1915.585229] LNet: Removed LNI 192.168.203.56@tcp [ 1915.596388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1915.634882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1915.654416] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1916.153947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1916.199164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1920.192800] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 22:05:56 (1767323156) [ 1920.365814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1920.375643] LNet: 114442:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1920.378789] LNetError: 114442:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1920.387985] LNet: Removed LNI 192.168.203.56@tcp [ 1920.397877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1920.422871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1920.439258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1920.851826] Key type .llcrypt unregistered [ 1920.854589] Key type ._llcrypt unregistered [ 1929.044991] Key type ._llcrypt registered [ 1929.047315] Key type .llcrypt registered [ 1930.108969] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 22:06:06 (1767323166) [ 1930.391940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1930.414837] alg: No test for adler32 (adler32-zlib) [ 1931.313864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1931.365419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1931.384279] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1931.982962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1932.046765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 1937.023718] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 22:06:13 (1767323173) [ 1937.174825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1937.198621] LNet: 115993:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1937.201937] LNetError: 115993:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1937.213395] LNet: Removed LNI 192.168.203.56@tcp [ 1937.230608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1937.282241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1937.304842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1941.980657] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 22:06:18 (1767323178) [ 1942.139701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1942.165161] LNet: 116573:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1942.182071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1942.257112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1942.594870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1942.623146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1942.665454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1942.935789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1942.965869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1943.024811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1943.052604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1943.102649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1943.374608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1943.408396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1943.479387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1943.796930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1943.829379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1948.580914] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 22:06:25 (1767323185) [ 1948.701215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1948.729609] LNet: 117199:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1948.738691] LNet: 117199:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1948.768033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1948.846460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1948.870721] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1949.401624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1949.424575] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 1950.093224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1950.136727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 1950.307962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1950.324962] LNetError: 117485:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1951.391250] LNet: Removed LNI 192.168.203.56@tcp [ 1951.393750] LNetError: 117485:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1951.407717] LNet: Removed LNI 192.168.203.56@tcp2 [ 1951.422717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1951.461713] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1951.465461] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 1951.471615] LNet: 117491:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1951.905679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 1952.092430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1952.114201] LNet: 117528:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1952.121978] LNet: 117528:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1953.186770] LNet: Removed LNI 192.168.203.56@tcp [ 1953.199406] LNet: Removed LNI 192.168.203.56@tcp2 [ 1953.226756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1953.276624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1953.298964] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1953.865583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1953.885306] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 1954.541665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 1954.568458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1954.611818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 1954.812525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1954.828540] LNetError: 117813:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1954.836126] LNetError: 117813:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1954.842818] LNet: Removed LNI 192.168.203.56@tcp [ 1955.879610] LNet: Removed LNI 192.168.203.56@tcp2 [ 1955.911707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1955.986719] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1955.997483] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 1956.008620] LNet: 117819:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1956.495291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 1956.690663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1956.704239] LNet: 117856:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1956.709227] LNet: 117856:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1956.713652] LNet: Removed LNI 192.168.203.56@tcp [ 1957.733227] LNet: Removed LNI 192.168.203.56@tcp2 [ 1957.766233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1957.849354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1957.876698] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1958.464888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1958.483297] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 1959.239661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1959.281811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 1959.510443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1959.531679] LNetError: 118135:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1959.537886] LNetError: 118135:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1959.543802] LNet: Removed LNI 192.168.203.56@tcp [ 1959.550741] LNet: Removed LNI 192.168.203.56@tcp2 [ 1959.568759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1959.616827] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1959.621815] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 1959.625585] LNet: 118141:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1960.162153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 1960.340572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1961.375441] LNet: Removed LNI 192.168.203.56@tcp [ 1961.408854] LNet: Removed LNI 192.168.203.56@tcp2 [ 1961.446724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1961.508459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1961.527244] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1962.150141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1962.165154] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 1962.880815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 1962.909700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1962.952514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 1963.130177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1964.192690] LNet: Removed LNI 192.168.203.56@tcp [ 1965.223360] LNet: Removed LNI 192.168.203.56@tcp2 [ 1965.269695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1965.344099] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1965.350280] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 1965.360820] LNet: 118469:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1965.684566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 1965.845706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1965.861097] LNet: 118506:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1965.863449] LNet: 118506:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1965.866634] LNet: Removed LNI 192.168.203.56@tcp [ 1966.886527] LNet: Removed LNI 192.168.203.56@tcp2 [ 1966.909681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1966.969566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1967.000898] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1967.441956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1967.459870] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 1967.997553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1968.022308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1968.086697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 1968.359210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1968.373257] LNetError: 118803:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1968.382241] LNetError: 118803:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1968.387505] LNet: Removed LNI 192.168.203.56@tcp [ 1968.400988] LNet: Removed LNI 192.168.203.56@tcp2 [ 1968.419431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1968.482093] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1968.487962] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 1968.848430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 1969.097079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1970.143175] LNet: Removed LNI 192.168.203.56@tcp [ 1970.156382] LNet: Removed LNI 192.168.203.56@tcp2 [ 1970.188848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1970.226507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1970.255454] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1970.881758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1970.899298] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 1971.566509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 1971.597707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1971.622891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1971.658591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 1971.923926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1972.959195] LNet: Removed LNI 192.168.203.56@tcp [ 1973.998172] LNet: Removed LNI 192.168.203.56@tcp2 [ 1974.026866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1974.084962] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1974.089606] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 1974.096350] LNet: 119169:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1974.099673] LNet: 119169:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 1974.531327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 1980.270981] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1981.354261] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 22:06:58 (1767323218) [ 1981.515649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1981.532266] LNet: Removed LNI 192.168.203.56@tcp [ 1982.565707] LNet: Removed LNI 192.168.203.56@tcp2 [ 1982.596372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1982.847564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1982.867045] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1983.301482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1983.317928] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1984.181939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1984.194805] LNet: 120201:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1984.198546] LNet: 120201:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1984.202711] LNet: Removed LNI 192.168.203.56@tcp [ 1984.212794] LNet: Removed LNI 10.1.2.3@tcp [ 1984.224188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1984.283733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1984.302362] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1984.309358] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1984.574471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1984.602954] LNet: Removed LNI 192.168.203.56@tcp [ 1984.616160] LNet: Removed LNI 10.1.2.3@tcp [ 1988.878641] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 22:07:05 (1767323225) [ 1988.939683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1988.972333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1989.057154] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1989.144450] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1989.147168] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1989.149145] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1991.397677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1991.421424] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1991.871562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1991.887136] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1993.196855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 1993.225186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1993.258959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.56@tcp -m PUT -e local_timeout -r 1 [ 1993.304504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1993.355141] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1993.356877] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1993.510962] LNetError: 120864:0:(lib-move.c:4947:lnet_parse()) 192.168.203.56@tcp, src 192.168.203.56@tcp: Dropping PUT (error -108 looking up sender) [ 1994.401584] LNetError: 121323:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1994.412155] LNetError: 121323:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 1994.425256] LNet: Removed LNI 192.168.203.56@tcp [ 1994.441547] LNet: Removed LNI 10.1.2.3@tcp [ 1994.807452] Key type .llcrypt unregistered [ 1994.810200] Key type ._llcrypt unregistered [ 2003.405195] Key type ._llcrypt registered [ 2003.407072] Key type .llcrypt registered [ 2004.247251] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 22:07:20 (1767323240) [ 2004.557245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2004.568072] alg: No test for adler32 (adler32-zlib) [ 2005.445487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2005.721926] LNet: 122172:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2006.128484] Key type .llcrypt unregistered [ 2006.130080] Key type ._llcrypt unregistered [ 2009.971243] Key type ._llcrypt registered [ 2009.972848] Key type .llcrypt registered [ 2010.065941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2010.075968] alg: No test for adler32 (adler32-zlib) [ 2010.926612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2010.941408] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2015.442557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 2015.576323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2015.907965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.156@tcp [ 2016.041579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2016.062214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2016.082846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2016.096203] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.156@tcp [ 2016.105461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2016.119731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2016.139588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2016.159373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2016.181398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2016.191665] LNet: 122617:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2016.204981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2016.226547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2016.247786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2016.291585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2016.325162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2016.344783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2016.736523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2016.879862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2016.898533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2016.916750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2016.935506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2016.957283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2016.982230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2017.008881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2017.033311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2017.045639] LNet: 122734:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2017.057527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2017.077853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2017.104118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2017.133556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2017.157703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2017.181692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2017.592827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.156@tcp [ 2017.750061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2017.765994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2017.797937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2017.816556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2017.838508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2017.855351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2017.890243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2017.911983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2017.937602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2017.967608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2017.989543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2018.004324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2018.018325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2018.040602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2018.474922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2018.642436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2018.669316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2018.702365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2018.732117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2018.761530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2018.779526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2018.795831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2018.817813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2018.830309] LNet: 122968:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2018.833330] LNet: 122968:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 2018.841554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2018.861857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2018.881058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2018.905880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2018.926268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2018.953428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2019.138779] LNet: 123013:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2019.150798] LNetError: 123013:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2019.164242] LNet: Removed LNI 192.168.203.56@tcp [ 2019.455597] Key type .llcrypt unregistered [ 2019.458435] Key type ._llcrypt unregistered [ 2030.144405] Key type ._llcrypt registered [ 2030.145670] Key type .llcrypt registered [ 2030.991716] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 22:07:47 (1767323267) [ 2031.238953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2031.247750] alg: No test for adler32 (adler32-zlib) [ 2032.099842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2032.320380] LNet: 123896:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2032.645333] Key type .llcrypt unregistered [ 2032.648762] Key type ._llcrypt unregistered [ 2036.392394] Key type ._llcrypt registered [ 2036.395851] Key type .llcrypt registered [ 2036.528257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2036.537367] alg: No test for adler32 (adler32-zlib) [ 2037.418383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2037.434858] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2042.799497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 2048.375849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2048.412843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2048.434036] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 2049.140518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2049.609134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.156@tcp [ 2049.759581] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.156@tcp [ 2049.792425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2049.823995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2049.845791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2049.868932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2049.895098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2049.927902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2049.962936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2050.000436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2050.026537] LNet: 124542:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2050.041904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2050.073779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2050.102093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2050.130945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2050.166314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2050.216586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2050.765668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2050.969605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2050.999815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2051.030594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2051.052401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2051.073454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2051.100522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2051.140575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2051.166818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2051.182457] LNet: 124660:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2051.199408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2051.222525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2051.260898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2051.290913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2051.332661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2051.373365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2051.970284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.156@tcp [ 2052.162844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2052.184292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2052.215938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2052.248706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2052.274481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2052.303377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2052.350196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2052.382721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2052.395602] LNet: 124777:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2052.414606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2052.441813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2052.479317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2052.500694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2052.528358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2052.552319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2052.968259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2053.156710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2053.190786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2053.217612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2053.240617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2053.266604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2053.288423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2053.313920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2053.338618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2053.364457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2053.399271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2053.423134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2053.443807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2053.468782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2053.498897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2057.407557] LNet: 124974:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2057.412738] LNetError: 124974:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2058.465373] LNet: Removed LNI 192.168.203.56@tcp [ 2058.469061] LNetError: 124974:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2058.496875] LNet: Removed LNI 192.168.203.56@tcp1 [ 2058.793653] Key type .llcrypt unregistered [ 2058.795292] Key type ._llcrypt unregistered [ 2069.465761] Key type ._llcrypt registered [ 2069.467479] Key type .llcrypt registered [ 2070.405334] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 22:08:26 (1767323306) [ 2074.278415] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 22:08:31 (1767323311) [ 2078.633859] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 22:08:35 (1767323315) [ 2083.212733] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 22:08:40 (1767323320) [ 2088.641305] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 22:08:45 (1767323325) [ 2093.507536] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 22:08:50 (1767323330) [ 2098.183875] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 22:08:54 (1767323334) [ 2103.271460] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 22:08:59 (1767323339) [ 2108.481870] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 22:09:05 (1767323345) [ 2113.395236] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 22:09:10 (1767323350) [ 2118.205896] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 22:09:15 (1767323355) [ 2123.650877] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 22:09:20 (1767323360) [ 2128.224162] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 22:09:24 (1767323364) [ 2132.545311] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 22:09:29 (1767323369) [ 2136.933582] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 22:09:33 (1767323373) [ 2141.265351] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 22:09:37 (1767323377) [ 2145.542327] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 22:09:42 (1767323382) [ 2149.881749] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 22:09:46 (1767323386) [ 2150.167843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2150.181297] alg: No test for adler32 (adler32-zlib) [ 2151.014903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2151.113392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2151.133323] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2151.872719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2151.885452] LNet: 135674:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2151.889718] LNetError: 135674:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2151.903754] LNet: Removed LNI 192.168.203.56@tcp [ 2151.915546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2151.954447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2151.969964] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2156.297961] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 22:09:53 (1767323393) [ 2160.948287] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 22:09:57 (1767323397) [ 2165.380428] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 22:10:02 (1767323402) [ 2169.537632] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 22:10:06 (1767323406) [ 2173.955147] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 22:10:10 (1767323410) [ 2178.664469] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 22:10:15 (1767323415) [ 2183.216899] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 22:10:19 (1767323419) [ 2187.878859] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 22:10:24 (1767323424) [ 2192.351627] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 22:10:29 (1767323429) [ 2197.555431] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 22:10:34 (1767323434) [ 2201.937741] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 22:10:38 (1767323438) [ 2206.397256] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 22:10:43 (1767323443) [ 2210.466694] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1901 sec ========== 22:10:47 (1767323447) [ 2210.733427] LNet: 143135:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2210.738391] LNetError: 143135:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2210.756944] LNet: Removed LNI 192.168.203.56@tcp [ 2211.180629] Key type .llcrypt unregistered [ 2211.182531] Key type ._llcrypt unregistered [ 2220.433265] Key type ._llcrypt registered [ 2220.435452] Key type .llcrypt registered [ 2220.663329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2220.679802] alg: No test for adler32 (adler32-zlib) [ 2221.641363] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g50ac801 [ 2221.825319] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2223.447433] Key type lgssc registered [ 2224.150642] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2298.589442] Lustre: Mounted lustre-client [ 2302.060304] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2310.940701] LustreError: 146284:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9827d104b800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 2310.952304] LustreError: 146284:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 2310.971148] Lustre: Unmounted lustre-client [ 2348.108388] Key type lgssc unregistered [ 2348.285432] LNet: 146853:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2348.290516] LNetError: 146853:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2348.304417] LNet: Removed LNI 192.168.203.56@tcp [ 2348.635179] Key type .llcrypt unregistered [ 2348.636948] Key type ._llcrypt unregistered