[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 477074729 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002320] x2apic enabled [ 0.003012] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008016] pid_max: default: 32768 minimum: 301 [ 0.009146] LSM: Security Framework initializing [ 0.011029] Yama: becoming mindful. [ 0.012023] SELinux: Initializing. [ 0.012943] *** VALIDATE selinux *** [ 0.020858] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024758] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025173] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026136] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027159] *** VALIDATE tmpfs *** [ 0.028548] *** VALIDATE proc *** [ 0.030177] *** VALIDATE cgroup *** [ 0.031014] *** VALIDATE cgroup2 *** [ 0.033058] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034165] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036041] Spectre V2 : User space: Vulnerable [ 0.037012] Speculative Store Bypass: Vulnerable [ 0.039953] debug: unmapping init [mem 0xffffffff86659000-0xffffffff86660fff] [ 0.041170] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042774] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043027] ... version: 2 [ 0.044015] ... bit width: 48 [ 0.045012] ... generic registers: 4 [ 0.045992] ... value mask: 0000ffffffffffff [ 0.046015] ... max period: 00007fffffffffff [ 0.047009] ... fixed-purpose events: 3 [ 0.047966] ... event mask: 000000070000000f [ 0.049187] rcu: Hierarchical SRCU implementation. [ 0.051212] smp: Bringing up secondary CPUs ... [ 0.052418] x86: Booting SMP configuration: [ 0.053030] .... node #0, CPUs: #1 #2 #3 [ 0.056731] smp: Brought up 1 node, 4 CPUs [ 0.058031] smpboot: Max logical packages: 1 [ 0.059036] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.280540] node 0 deferred pages initialised in 218ms [ 0.284097] devtmpfs: initialized [ 0.285366] x86/mm: Memory block size: 128MB [ 0.289000] gcov: version magic: 0x41383552 [ 0.291399] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.292098] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.293301] pinctrl core: initialized pinctrl subsystem [ 0.294208] [ 0.294578] ************************************************************* [ 0.295018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.296014] ** ** [ 0.297022] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.298019] ** ** [ 0.299019] ** This means that this kernel is built to expose internal ** [ 0.300016] ** IOMMU data structures, which may compromise security on ** [ 0.301015] ** your system. ** [ 0.302017] ** ** [ 0.303020] ** If you see this message and you are not debugging the ** [ 0.304012] ** kernel, report this immediately to your vendor! ** [ 0.305014] ** ** [ 0.306015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.307014] ************************************************************* [ 0.308953] NET: Registered protocol family 16 [ 0.310439] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.314083] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.317072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.321156] cpuidle: using governor menu [ 0.322742] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.325620] PCI: Using configuration type 1 for base access [ 0.328165] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.337206] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.340054] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.344072] cryptd: max_cpu_qlen set to 1000 [ 0.346378] ACPI: Added _OSI(Module Device) [ 0.347021] ACPI: Added _OSI(Processor Device) [ 0.348026] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.349016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.353000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.360621] ACPI: Interpreter enabled [ 0.361076] ACPI: PM: (supports S0 S3 S4 S5) [ 0.363038] ACPI: Using IOAPIC for interrupt routing [ 0.365243] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.369541] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.381320] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.384084] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.387034] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.390170] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.395537] acpiphp: Slot [2] registered [ 0.398204] acpiphp: Slot [5] registered [ 0.399140] acpiphp: Slot [6] registered [ 0.400141] acpiphp: Slot [3] registered [ 0.402174] acpiphp: Slot [4] registered [ 0.404156] acpiphp: Slot [7] registered [ 0.405132] acpiphp: Slot [8] registered [ 0.407163] acpiphp: Slot [9] registered [ 0.408160] acpiphp: Slot [10] registered [ 0.410154] acpiphp: Slot [11] registered [ 0.411190] acpiphp: Slot [12] registered [ 0.413146] acpiphp: Slot [13] registered [ 0.414117] acpiphp: Slot [14] registered [ 0.416133] acpiphp: Slot [15] registered [ 0.418125] acpiphp: Slot [16] registered [ 0.419151] acpiphp: Slot [17] registered [ 0.420126] acpiphp: Slot [18] registered [ 0.422210] acpiphp: Slot [19] registered [ 0.423177] acpiphp: Slot [20] registered [ 0.425143] acpiphp: Slot [21] registered [ 0.427164] acpiphp: Slot [22] registered [ 0.428143] acpiphp: Slot [23] registered [ 0.429138] acpiphp: Slot [24] registered [ 0.430209] acpiphp: Slot [25] registered [ 0.432142] acpiphp: Slot [26] registered [ 0.433144] acpiphp: Slot [27] registered [ 0.435146] acpiphp: Slot [28] registered [ 0.436180] acpiphp: Slot [29] registered [ 0.438161] acpiphp: Slot [30] registered [ 0.439135] acpiphp: Slot [31] registered [ 0.441091] PCI host bridge to bus 0000:00 [ 0.443034] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.445041] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.447042] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.450034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.452036] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.455049] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.457213] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.460609] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.463420] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.471019] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.477011] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.479027] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.481027] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.483027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.485345] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.487806] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.490056] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.493918] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.498025] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.508023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.512016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.517188] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.541033] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.557031] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.578037] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.589743] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.601026] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.608023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.627029] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.637952] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.641499] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.644388] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.646417] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.648336] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.653176] iommu: Default domain type: Passthrough [ 0.655474] SCSI subsystem initialized [ 0.657314] ACPI: bus type USB registered [ 0.659135] usbcore: registered new interface driver usbfs [ 0.661131] usbcore: registered new interface driver hub [ 0.662092] usbcore: registered new device driver usb [ 0.664216] pps_core: LinuxPPS API ver. 1 registered [ 0.666022] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.670083] PTP clock support registered [ 0.673115] EDAC MC: Ver: 3.0.0 [ 0.675188] PCI: Using ACPI for IRQ routing [ 0.677996] NetLabel: Initializing [ 0.679017] NetLabel: domain hash size = 128 [ 0.680014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.682179] NetLabel: unlabeled traffic allowed by default [ 0.685065] vgaarb: loaded [ 0.687307] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.689023] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.698156] clocksource: Switched to clocksource kvm-clock [ 0.820880] VFS: Disk quotas dquot_6.6.0 [ 0.822533] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.825328] *** VALIDATE ramfs *** [ 0.826598] *** VALIDATE hugetlbfs *** [ 0.828328] pnp: PnP ACPI init [ 0.831052] pnp: PnP ACPI: found 6 devices [ 0.852529] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.855459] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.857479] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.859297] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.861264] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.863145] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.864977] NET: Registered protocol family 2 [ 0.867527] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.872798] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.875597] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.879839] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.882297] TCP: Hash tables configured (established 65536 bind 65536) [ 0.884563] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.886609] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.888582] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.890558] NET: Registered protocol family 1 [ 0.892359] RPC: Registered named UNIX socket transport module. [ 0.893886] RPC: Registered udp transport module. [ 0.895233] RPC: Registered tcp transport module. [ 0.896288] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.897624] NET: Registered protocol family 44 [ 0.898989] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.900368] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.901740] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.903189] PCI: CLS 0 bytes, default 64 [ 0.904398] Unpacking initramfs... [ 2.354483] debug: unmapping init [mem 0xffff9044fcc64000-0xffff9044fffcffff] [ 2.359849] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.362405] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.365628] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.897395] Initialise system trusted keyrings [ 2.899119] Key type blacklist registered [ 2.900789] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.909823] zbud: loaded [ 2.913100] *** VALIDATE nfs *** [ 2.914362] *** VALIDATE nfs4 *** [ 2.915952] pstore: using deflate compression [ 2.918547] Platform Keyring initialized [ 3.028554] NET: Registered protocol family 38 [ 3.030187] Key type asymmetric registered [ 3.031423] Asymmetric key parser 'x509' registered [ 3.033377] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.037130] io scheduler mq-deadline registered [ 3.039122] io scheduler kyber registered [ 3.040559] io scheduler bfq registered [ 3.042538] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.044985] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.048168] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.050649] ACPI: Power Button [PWRF] [ 3.055733] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.061133] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.070926] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.098612] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.127729] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.133251] Non-volatile memory driver v1.3 [ 3.134945] Linux agpgart interface v0.103 [ 3.164423] virtio_blk virtio1: [vda] 146624 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.167295] vda: detected capacity change from 0 to 75071488 [ 3.186817] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.190012] vdb: detected capacity change from 0 to 1073741824 [ 3.197120] libphy: Fixed MDIO Bus: probed [ 3.212900] usbcore: registered new interface driver usbserial_generic [ 3.215833] usbserial: USB Serial support registered for generic [ 3.218232] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.222631] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.223941] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.225741] mousedev: PS/2 mouse device common for all mice [ 3.229086] rtc_cmos 00:05: RTC can wake from S4 [ 3.232186] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.233109] rtc_cmos 00:05: registered as rtc0 [ 3.236386] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.238454] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.239920] intel_pstate: CPU model not supported [ 3.244164] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.247248] hid: raw HID events driver (C) Jiri Kosina [ 3.249307] usbcore: registered new interface driver usbhid [ 3.250507] usbhid: USB HID core driver [ 3.251388] drop_monitor: Initializing network drop monitor service [ 3.252754] Initializing XFRM netlink socket [ 3.253866] NET: Registered protocol family 10 [ 3.256695] Segment Routing with IPv6 [ 3.258058] NET: Registered protocol family 17 [ 3.259672] mpls_gso: MPLS GSO support [ 3.264692] RAS: Correctable Errors collector initialized. [ 3.265991] AVX version of gcm_enc/dec engaged. [ 3.266831] AES CTR mode by8 optimization enabled [ 3.342592] sched_clock: Marking stable (3342563810, 0)->(4229051149, -886487339) [ 3.346670] registered taskstats version 1 [ 3.348910] Loading compiled-in X.509 certificates [ 3.350543] zswap: loaded using pool lzo/zbud [ 3.375677] Key type big_key registered [ 3.389932] Key type encrypted registered [ 3.391867] ima: No TPM chip found, activating TPM-bypass! [ 3.393923] ima: Allocated hash algorithm: sha1 [ 3.395539] ima: No architecture policies found [ 3.397285] evm: Initialising EVM extended attributes: [ 3.398901] evm: security.selinux [ 3.399763] evm: security.ima [ 3.400383] evm: security.capability [ 3.401298] evm: HMAC attrs: 0x1 [ 3.402936] rtc_cmos 00:05: setting system clock to 2026-09-04 00:45:15 UTC (1788482715) [ 3.407460] debug: unmapping init [mem 0xffffffff87603000-0xffffffff877fffff] [ 3.409308] debug: unmapping init [mem 0xffffffff86382000-0xffffffff86658fff] [ 3.419173] Write protecting the kernel read-only data: 28672k [ 3.423200] debug: unmapping init [mem 0xffffffff84a03000-0xffffffff84bfffff] [ 3.425907] debug: unmapping init [mem 0xffffffff85314000-0xffffffff853fffff] [ 3.460656] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.467646] systemd[1]: Detected virtualization kvm. [ 3.469683] systemd[1]: Detected architecture x86-64. [ 3.471833] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.495848] systemd[1]: No hostname configured. [ 3.497378] systemd[1]: Set hostname to . [ 3.498615] random: systemd: uninitialized urandom read (16 bytes read) [ 3.501111] systemd[1]: Initializing machine ID from random generator. [ 3.537292] random: ln: uninitialized urandom read (6 bytes read) [ 3.643163] random: systemd: uninitialized urandom read (16 bytes read) [ 3.644840] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.648827] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.652225] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Reached target Timers. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.286328] device-mapper: uevent: version 1.0.3 [ 4.288520] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager...[ 4.552714] random: fast init done [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.072149] virtio_net virtio0 ens2: renamed from eth0 [ 5.108520] scsi host0: ata_piix [ 5.146403] scsi host1: ata_piix [ 5.148362] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.150958] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.887683] dracut-initqueue[577]: RTNETLINK answers: File exists [ 9.904844] random: crng init done [ 9.906710] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.350801] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.412451] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.657550] SELinux: Disabled at runtime. [ 11.714701] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.724698] systemd[1]: Detected virtualization kvm. [ 11.726672] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.209279] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.212669] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.217517] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.222406] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.224701] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.230455] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.245352] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. Mounting Huge Pages File System... Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. [ 12.357411] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.601268] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.880529] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.918365] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.984112] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.014990] EDAC sbridge: Ver: 1.1.2 [ 13.915451] Key type dns_resolver registered [ 14.225321] NFS: Registering the id_resolver key type [ 14.227295] Key type id_resolver registered [ 14.228968] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg105-client login: [ 75.802310] libcfs: loading out-of-tree module taints kernel. [ 75.986959] Key type ._llcrypt registered [ 75.990407] Key type .llcrypt registered [ 76.623830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 76.641795] alg: No test for adler32 (adler32-zlib) [ 78.215362] Lustre: Lustre: Build Version: 2.17.57_84_g5b8c505 [ 79.062924] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 80.808211] Key type lgssc registered [ 82.557631] Lustre: Echo OBD driver; http://www.lustre.org/ [ 214.179278] hrtimer: interrupt took 5032413 ns [ 228.825236] Lustre: Mounted lustre-client - version 2.17.57_84_g5b8c505 [ 234.095869] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 247.096163] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing check_logdir /tmp/testlogs/ [ 252.292713] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing yml_node [ 254.435188] Lustre: lustre-OST0000-osc-ffff904549a7a800: disconnect after 23s idle [ 257.748771] Lustre: DEBUG MARKER: Client: 2.17.57.84 [ 260.484625] Lustre: DEBUG MARKER: MDS: 2.17.57.84 [ 263.341403] Lustre: DEBUG MARKER: OSS: 2.17.57.84 [ 265.188677] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Sep 3 20:49:35 EDT 2026 [ 281.889484] Lustre: Unmounted lustre-client [ 310.059315] Key type lgssc unregistered [ 310.444598] LNet: 8407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 310.452992] LNetError: 8407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 311.527297] LNet: Removed LNI 192.168.201.5@tcp [ 312.328312] Key type .llcrypt unregistered [ 312.333918] Key type ._llcrypt unregistered [ 341.339432] Key type ._llcrypt registered [ 341.341708] Key type .llcrypt registered [ 341.498553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 341.513430] alg: No test for adler32 (adler32-zlib) [ 342.448215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 342.545370] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 342.583130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 346.852300] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 347.005406] LNet: 9281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 347.014807] LNetError: 9281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 347.064210] LNet: Removed LNI 192.168.201.5@tcp [ 347.829509] Key type .llcrypt unregistered [ 347.831175] Key type ._llcrypt unregistered [ 354.818753] Key type ._llcrypt registered [ 354.821823] Key type .llcrypt registered [ 356.517443] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 20:51:06 (1788483066) [ 356.973291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 357.001858] alg: No test for adler32 (adler32-zlib) [ 358.017552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 358.107530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 358.134945] LNet: 9696:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 358.158483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 358.217882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 358.238420] LNet: 9703:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 365.775109] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 20:51:16 (1788483076) [ 365.934394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 365.951234] LNet: 10278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 365.985979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 366.115865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 373.011585] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 20:51:23 (1788483083) [ 373.258496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 373.311507] LNet: 10868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 373.378523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 373.473433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 379.998909] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 20:51:30 (1788483090) [ 380.199714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.225595] LNet: 11452:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 380.241155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.311502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 386.101627] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 20:51:36 (1788483096) [ 386.296901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 386.315717] LNet: 12037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 386.336807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 386.410941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 386.763687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 393.160793] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 20:51:43 (1788483103) [ 393.314606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 393.365040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.462628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 393.828053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 394.279430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 400.663403] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 20:51:51 (1788483111) [ 401.015598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.034278] LNet: 13225:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 401.036966] LNet: 13225:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 401.060658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.115437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 407.292345] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 20:51:58 (1788483118) [ 407.619180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 407.660490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 407.712438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 407.752937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 408.165361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 408.196358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 408.720251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 408.790449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 409.105269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 409.130801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 409.626480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 409.675823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 410.084721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 410.118692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 410.609864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 410.643967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 410.983846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 411.022312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 411.390854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 411.455266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 411.954501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 411.999330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 412.451574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 412.489885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 412.931803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 412.971692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 413.480697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 413.538678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 419.414357] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 20:52:10 (1788483130) [ 419.578456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 419.593270] LNet: 14491:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 419.596925] LNet: 14491:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 419.635547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 419.748182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 419.780889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 426.173899] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 20:52:17 (1788483137) [ 426.618632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 426.688813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 426.804603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 426.853778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 434.884622] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 20:52:25 (1788483145) [ 435.164245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.247955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.315888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 435.339420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 444.264460] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 20:52:34 (1788483154) [ 444.413895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 444.463280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 444.574395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 444.601834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 452.931299] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 20:52:43 (1788483163) [ 453.042715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.082490] LNet: 16829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 453.091337] LNet: 16829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 453.141694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.317313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 453.359305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 460.403608] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 20:52:51 (1788483171) [ 460.767817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.819369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.893217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 460.933911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 467.742763] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 20:52:58 (1788483178) [ 468.160357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.206490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.366670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 468.879193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 475.959765] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 20:53:06 (1788483186) [ 476.253388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.320896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.580617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 476.896682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 485.427623] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 20:53:15 (1788483195) [ 485.557743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.625853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.881945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 486.129502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 494.467253] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 20:53:25 (1788483205) [ 494.808420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 494.845726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 495.187781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 495.337546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 506.768410] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 20:53:36 (1788483216) [ 506.978536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 507.071714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.233914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 507.283247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 516.811847] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 20:53:47 (1788483227) [ 517.279691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 517.306032] LNet: 21084:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 517.313856] LNet: 21084:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 517.352924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 517.471242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 517.550935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 526.420363] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 20:53:56 (1788483236) [ 526.820022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 526.906335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 527.052621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 535.590756] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 20:54:06 (1788483246) [ 536.126992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 536.172649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 536.281681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 544.208597] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 20:54:14 (1788483254) [ 544.480124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 544.539155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 544.663481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 545.175023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 552.634616] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 20:54:23 (1788483263) [ 552.930437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 552.986901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 553.104229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 553.168839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 560.243573] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 20:54:31 (1788483271) [ 560.492876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 560.553449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 560.685581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 560.717856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 567.011221] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 20:54:38 (1788483278) [ 567.383274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 567.448375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 567.524408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 567.570361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 575.183466] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 20:54:45 (1788483285) [ 575.637686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 575.708295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 575.809675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 575.852373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 583.050658] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 20:54:53 (1788483293) [ 583.366871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 583.410663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.468824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 583.559937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 583.580843] LNetError: 25810:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 583.628453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 583.663335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 589.840603] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 20:55:00 (1788483300) [ 589.983945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 590.075084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 590.168353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.T1i3vs0HP7 [ 590.271418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.T1i3vs0HP7 [ 590.318704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.T1i3vs0HP7 [ 597.353326] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 20:55:08 (1788483308) [ 597.585802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 597.632947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 597.687735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 597.727937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 597.784954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 604.296878] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 20:55:14 (1788483314) [ 604.405657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 604.504193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.621744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 604.636890] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 606.589113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.5@tcp [ 608.095814] Lustre: Lustre: Build Version: 2.17.57_84_g5b8c505 [ 610.105048] Key type lgssc registered [ 611.616039] Lustre: Echo OBD driver; http://www.lustre.org/ [ 616.543724] LNetError: 27570:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.201.5@tcp99 has no available nets [ 619.441170] Key type lgssc unregistered [ 619.786627] LNetError: 28678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 619.804048] LNet: Removed LNI 192.168.201.5@tcp [ 620.440106] Key type .llcrypt unregistered [ 620.442329] Key type ._llcrypt unregistered [ 628.090133] Key type ._llcrypt registered [ 628.093944] Key type .llcrypt registered [ 629.305367] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 20:55:40 (1788483340) [ 630.077328] Key type .llcrypt unregistered [ 630.079137] Key type ._llcrypt unregistered [ 637.718893] Key type ._llcrypt registered [ 637.724113] Key type .llcrypt registered [ 637.941305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 637.964534] alg: No test for adler32 (adler32-zlib) [ 638.980872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.106048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 639.130242] LNet: 29589:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 639.138685] LNet: 29589:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 639.148147] LNet: 29589:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 639.169159] LNet: Message forwarding will be enabled in 2s [ 642.272256] LNet: Message forwarding enabled [ 642.765912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 642.780229] LNet: Message forwarding will stop in 2s to 6s [ 646.371355] LNet: Message forwarding disabled [ 647.409820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 647.427986] LNet: 29688:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 647.444330] LNet: 29688:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 647.458051] LNet: 29688:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 647.478423] LNet: Message forwarding will be enabled in 2s [ 647.673631] LNet: 29701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 648.495586] Key type .llcrypt unregistered [ 648.498280] Key type ._llcrypt unregistered [ 661.383950] Key type ._llcrypt registered [ 661.386841] Key type .llcrypt registered [ 662.893431] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 20:56:13 (1788483373) [ 663.329795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 663.340859] alg: No test for adler32 (adler32-zlib) [ 664.251739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.318263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 671.429709] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 20:56:22 (1788483382) [ 671.672859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.715226] LNet: 31099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 671.794641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.909267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 671.979223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 672.064124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 672.146450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 672.219719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 672.279064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 672.365232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 672.445938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 672.505783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 672.580549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 672.617869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 672.654802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 672.706757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 672.800849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 672.895133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 672.972449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 673.025973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 673.053046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 673.088659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 673.105241] LNetError: 31144:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 679.314123] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 20:56:30 (1788483390) [ 679.706817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 679.745805] LNet: 31716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 679.773846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 679.887471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 686.410688] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 20:56:37 (1788483397) [ 688.210151] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg105-server [ 689.906721] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 20:56:40 (1788483400) [ 690.172846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 690.202487] LNet: 32648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 690.215243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.260900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 690.292365] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 691.143811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[6-10]@tcp [ 691.179140] LNet: 32781:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 691.200443] LNetError: 32656:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.7@tcp has gone from up to down [ 691.215619] LNetError: 32652:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.6@tcp is being used as a gateway but routing feature is not turned on [ 698.009762] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 20:56:48 (1788483408) [ 699.817689] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg105-server [ 702.069616] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 20:56:52 (1788483412) [ 702.223367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 702.250840] LNet: 33714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 702.262796] LNetError: 33714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 702.303362] LNet: Removed LNI 192.168.201.5@tcp [ 702.342266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 702.390758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 702.405387] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 703.121644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[6-11/2]@tcp [ 703.142621] LNet: 33835:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 703.149820] LNet: 33835:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 703.174604] LNetError: 33722:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.6@tcp is being used as a gateway but routing feature is not turned on [ 703.204532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[6-11/2]@tcp [ 703.205666] LNetError: 33722:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 703.205703] LNetError: 33722:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.6@tcp has gone from up to down [ 703.205709] LNetError: 33722:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 710.852509] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 20:57:01 (1788483421) [ 711.132772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 711.158164] LNet: 34427:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 711.163668] LNetError: 34427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 711.192695] LNet: Removed LNI 192.168.201.5@tcp [ 711.228625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.356631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 711.410370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 711.464935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.556421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 711.633938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 711.674518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.721883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 711.808523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 711.837951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.900894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 712.025972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.082394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.163961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 712.230831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.253798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.303223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 712.364838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.394340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.458994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 712.542962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.577963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.636949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 712.726796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.772706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.851042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 713.006637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.100429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.168343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 713.253554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.285981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.342744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 713.415414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.474942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.594907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 720.408488] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 20:57:10 (1788483430) [ 722.820480] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg105-server [ 725.132246] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 20:57:15 (1788483435) [ 727.125693] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg105-server [ 728.954449] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 20:57:19 (1788483439) [ 729.283850] LNet: 35828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 729.294329] LNet: 35828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 730.042870] Key type .llcrypt unregistered [ 730.049576] Key type ._llcrypt unregistered [ 737.070860] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 737.242697] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 737.254689] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 737.266141] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 740.041301] Key type ._llcrypt registered [ 740.043619] Key type .llcrypt registered [ 740.230622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 740.250622] alg: No test for adler32 (adler32-zlib) [ 741.201968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 741.254659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 741.272725] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 741.886634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 741.906812] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 743.845592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 743.882324] LNet: Removed LNI 10.1.2.3@tcp [ 744.005793] LNet: 36544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 744.010567] LNetError: 36544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 744.026586] LNet: Removed LNI 192.168.201.5@tcp [ 744.717624] Key type .llcrypt unregistered [ 744.724215] Key type ._llcrypt unregistered [ 756.053717] Key type ._llcrypt registered [ 756.055806] Key type .llcrypt registered [ 757.370910] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 20:57:48 (1788483468) [ 757.940448] Key type .llcrypt unregistered [ 757.944373] Key type ._llcrypt unregistered [ 766.213134] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 766.395986] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 766.404139] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 766.412112] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 769.354107] Key type ._llcrypt registered [ 769.360257] Key type .llcrypt registered [ 769.791931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 769.810456] alg: No test for adler32 (adler32-zlib) [ 770.876755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 770.931765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 770.950032] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 771.883969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 771.895700] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 774.512862] LNet: 38035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 774.519696] LNetError: 38035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 774.524538] LNet: Removed LNI 10.1.2.3@tcp [ 774.535331] LNet: Removed LNI 192.168.201.5@tcp [ 775.109657] Key type .llcrypt unregistered [ 775.111583] Key type ._llcrypt unregistered [ 789.143692] Key type ._llcrypt registered [ 789.146706] Key type .llcrypt registered [ 790.653653] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 20:58:20 (1788483500) [ 791.169357] Key type .llcrypt unregistered [ 791.172390] Key type ._llcrypt unregistered [ 798.042978] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 798.211762] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 798.235938] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 798.241669] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 801.146889] Key type ._llcrypt registered [ 801.152684] Key type .llcrypt registered [ 801.386544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 801.404966] alg: No test for adler32 (adler32-zlib) [ 802.414649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 802.491295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 802.524883] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 804.729733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 804.749848] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 806.355908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 806.368278] LNet: Removed LNI 10.1.2.3@tcp [ 806.383799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 806.412806] LNet: Removed LNI 10.1.2.31@tcp [ 806.427444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 806.471412] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 807.276445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 807.294597] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 808.151510] LNet: 39949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 808.165111] LNetError: 39949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 808.171975] LNet: Removed LNI 10.1.2.31@tcp [ 808.184941] LNet: Removed LNI 10.1.2.3@tcp [ 808.953589] Key type .llcrypt unregistered [ 808.956630] Key type ._llcrypt unregistered [ 823.347701] Key type ._llcrypt registered [ 823.350344] Key type .llcrypt registered [ 824.910932] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 20:58:55 (1788483535) [ 825.595160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 825.625613] alg: No test for adler32 (adler32-zlib) [ 826.630760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 826.703958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 826.740989] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 827.610634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 827.624222] LNet: 40899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 827.627795] LNetError: 40899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 827.647902] LNet: Removed LNI 192.168.201.5@tcp [ 827.669217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 827.738919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.5@tcp [ 827.766590] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 828.449422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.5@tcp [ 828.474071] LNet: Removed LNI 192.168.201.5@tcp [ 828.528920] LNet: 41034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 829.153654] Key type .llcrypt unregistered [ 829.155326] Key type ._llcrypt unregistered [ 842.020340] Key type ._llcrypt registered [ 842.022481] Key type .llcrypt registered [ 843.555923] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 20:59:13 (1788483553) [ 845.012221] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg105-server [ 847.559611] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 20:59:17 (1788483557) [ 848.436631] Key type .llcrypt unregistered [ 848.441445] Key type ._llcrypt unregistered [ 856.928817] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 857.068791] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 857.074446] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 857.080836] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 859.841553] Key type ._llcrypt registered [ 859.845517] Key type .llcrypt registered [ 860.132067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 860.149732] alg: No test for adler32 (adler32-zlib) [ 861.178664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 861.320131] LNet: Added LNI 192.168.201.5@tcp1 [8/256/0/180] [ 862.499862] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 865.574779] LNet: 42860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 865.582425] LNetError: 42860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 865.592968] LNet: Removed LNI 192.168.201.5@tcp1 [ 865.613622] LNet: Removed LNI 10.1.2.3@tcp2 [ 866.211531] Key type .llcrypt unregistered [ 866.214339] Key type ._llcrypt unregistered [ 879.431993] Key type ._llcrypt registered [ 879.440671] Key type .llcrypt registered [ 881.254300] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 20:59:51 (1788483591) [ 881.598395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 881.621997] alg: No test for adler32 (adler32-zlib) [ 882.602769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.643539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 882.667732] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 883.384943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 883.402963] LNet: 43827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.408615] LNetError: 43827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.417951] LNet: Removed LNI 192.168.201.5@tcp [ 883.435303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.487914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 883.505117] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 891.047752] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 21:00:01 (1788483601) [ 891.422768] LNet: 44418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 891.442282] LNetError: 44418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 891.462032] LNet: Removed LNI 192.168.201.5@tcp [ 892.104733] Key type .llcrypt unregistered [ 892.110931] Key type ._llcrypt unregistered [ 898.650738] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 898.816460] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 898.829505] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 898.832208] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 900.715901] Key type ._llcrypt registered [ 900.718891] Key type .llcrypt registered [ 900.968099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 901.010381] alg: No test for adler32 (adler32-zlib) [ 902.079929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 902.189767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 902.244320] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 903.387686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 903.405562] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 906.228297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 906.242049] LNet: 45149:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 906.245601] LNetError: 45149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 906.248916] LNet: Removed LNI 10.1.2.3@tcp [ 906.264802] LNet: Removed LNI 192.168.201.5@tcp [ 906.321288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 906.420162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 906.455096] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 906.459147] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 906.764370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 906.784383] LNet: 45176:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 906.791668] LNetError: 45176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 906.797290] LNetError: 45176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 906.814106] LNet: Removed LNI 10.1.2.3@tcp [ 906.826770] LNet: Removed LNI 192.168.201.5@tcp [ 906.880768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 906.988835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 907.041194] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 907.044809] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 907.647217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 907.658615] LNet: Added LNI 192.168.201.5@tcp2 [8/256/0/180] [ 908.829533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 908.880600] LNet: 45339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 908.889936] LNetError: 45339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 908.902461] LNetError: 45339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 908.911487] LNet: Removed LNI 10.1.2.3@tcp [ 908.915699] LNet: Removed LNI 192.168.201.5@tcp [ 908.926677] LNet: Removed LNI 192.168.201.5@tcp2 [ 908.978299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 909.196995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 909.275290] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 909.293861] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 909.312624] LNet: Added LNI 192.168.201.5@tcp2 [8/256/0/180] [ 920.987905] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 21:00:31 (1788483631) [ 921.515235] LNet: 45938:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.527364] LNetError: 45938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 921.540319] LNetError: 45938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 921.557130] LNet: Removed LNI 10.1.2.3@tcp [ 921.562392] LNet: Removed LNI 192.168.201.5@tcp [ 921.592249] LNet: Removed LNI 192.168.201.5@tcp2 [ 922.157562] Key type .llcrypt unregistered [ 922.159530] Key type ._llcrypt unregistered [ 933.035616] Key type ._llcrypt registered [ 933.038949] Key type .llcrypt registered [ 933.177804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 933.200688] alg: No test for adler32 (adler32-zlib) [ 934.161187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 934.268397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 934.301315] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 935.370783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 935.394768] LNet: 46331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 935.399988] LNetError: 46331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 935.414494] LNet: Removed LNI 192.168.201.5@tcp [ 935.426744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 935.503375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 935.544248] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 936.277073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 936.323677] LNet: 46361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 936.336807] LNetError: 46361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 936.358110] LNet: Removed LNI 192.168.201.5@tcp [ 936.400482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 936.498671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 936.543702] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 937.560431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 937.591279] LNet: 46391:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 937.604392] LNetError: 46391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 937.614565] LNet: Removed LNI 192.168.201.5@tcp [ 937.645119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 937.748506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 937.816408] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 938.316340] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 938.429656] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 938.431572] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 938.435322] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 939.971803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 940.011317] LNet: 46461:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 940.034380] LNetError: 46461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 940.060774] LNet: Removed LNI 192.168.201.5@tcp [ 940.098478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 940.142194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 940.161234] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 947.846362] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 21:00:58 (1788483658) [ 948.156640] LNet: 47057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.173105] LNetError: 47057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 948.216059] LNet: Removed LNI 192.168.201.5@tcp [ 949.107938] Key type .llcrypt unregistered [ 949.111782] Key type ._llcrypt unregistered [ 958.337290] Key type ._llcrypt registered [ 958.353198] Key type .llcrypt registered [ 958.655790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 958.691977] alg: No test for adler32 (adler32-zlib) [ 959.738306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 959.823726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 959.864074] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 961.046520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 961.101221] LNet: 47468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 961.110388] LNetError: 47468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 961.128897] LNet: Removed LNI 192.168.201.5@tcp [ 961.173196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 961.242831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 961.267582] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 969.964417] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 21:01:20 (1788483680) [ 970.122464] LNet: 48060:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.125321] LNetError: 48060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 970.133834] LNet: Removed LNI 192.168.201.5@tcp [ 970.570735] Key type .llcrypt unregistered [ 970.573623] Key type ._llcrypt unregistered [ 978.877093] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 978.946221] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 978.954303] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 978.967030] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 981.712797] Key type ._llcrypt registered [ 981.714921] Key type .llcrypt registered [ 981.975058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 982.010543] alg: No test for adler32 (adler32-zlib) [ 982.961791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 983.019571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 983.056253] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 984.023387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 984.042407] LNet: 48506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 984.054687] LNetError: 48506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 984.075860] LNet: Removed LNI 192.168.201.5@tcp [ 984.109421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 984.175827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 984.224071] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 991.682978] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 21:01:42 (1788483702) [ 992.104177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 992.123769] LNet: 49104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 992.128515] LNetError: 49104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 992.140076] LNet: Removed LNI 192.168.201.5@tcp [ 992.181391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 992.333648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 999.887491] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 21:01:50 (1788483710) [ 1000.254207] LNet: 49686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1001.073937] Key type .llcrypt unregistered [ 1001.078505] Key type ._llcrypt unregistered [ 1007.857819] Key type ._llcrypt registered [ 1007.861377] Key type .llcrypt registered [ 1008.089174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1008.107510] alg: No test for adler32 (adler32-zlib) [ 1009.032310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1009.114583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1009.136787] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 1010.344593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1010.378484] LNet: 50081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1010.394881] LNetError: 50081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1010.423022] LNet: Removed LNI 192.168.201.5@tcp [ 1010.447780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1010.576624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1010.624443] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 1019.191775] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 21:02:09 (1788483729) [ 1019.623325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1019.659632] LNet: 50675:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1019.674385] LNetError: 50675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1019.700971] LNet: Removed LNI 192.168.201.5@tcp [ 1019.747918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1019.874907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1027.547633] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 21:02:17 (1788483737) [ 1029.459418] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1031.932506] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 21:02:22 (1788483742) [ 1033.715314] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1035.729271] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 21:02:26 (1788483746) [ 1037.631606] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1039.031598] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 21:02:30 (1788483750) [ 1040.514259] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1042.578192] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 21:02:33 (1788483753) [ 1044.266379] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1045.976990] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 21:02:36 (1788483756) [ 1046.255798] LNet: 52730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1046.776626] Key type .llcrypt unregistered [ 1046.778557] Key type ._llcrypt unregistered [ 1053.870734] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1054.099738] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1054.132185] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1054.142411] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1056.880890] Key type ._llcrypt registered [ 1056.884854] Key type .llcrypt registered [ 1057.151591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1057.167631] alg: No test for adler32 (adler32-zlib) [ 1058.144127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1058.345793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1058.433034] LNet: Added LNI 192.168.201.5@tcp [8/512/0/180] [ 1058.441674] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1058.445214] LNet: 53047:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1058.535826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1058.551326] LNet: 53066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1058.556109] LNetError: 53066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1058.561141] LNet: Removed LNI 10.1.2.3@tcp [ 1058.563675] LNetError: 53066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1058.574201] LNet: Removed LNI 192.168.201.5@tcp [ 1058.595502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1058.736746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1058.774630] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 1058.874226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1058.890492] LNet: 53090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1058.895309] LNetError: 53090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1058.917576] LNet: Removed LNI 192.168.201.5@tcp [ 1058.943262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1059.040274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1059.058043] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 1059.065610] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1059.067493] LNet: 53098:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1059.200613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1059.239330] LNet: 53116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1059.259600] LNetError: 53116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1059.280195] LNet: Removed LNI 10.1.2.3@tcp [ 1059.286898] LNetError: 53116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1059.308826] LNet: Removed LNI 192.168.201.5@tcp [ 1059.351735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1059.425891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1059.461864] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 1059.468686] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1059.478053] LNet: 53124:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1059.673880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1059.720395] LNet: 53150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1059.748932] LNetError: 53150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1059.753054] LNet: Removed LNI 10.1.2.3@tcp [ 1059.754940] LNetError: 53150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1059.774422] LNet: Removed LNI 192.168.201.5@tcp [ 1059.788717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1059.872704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1059.895885] LNet: Added LNI 192.168.201.5@tcp [8/256/0/182] [ 1059.901119] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1060.010619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1060.048339] LNet: 53178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1060.057493] LNetError: 53178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1060.066850] LNet: Removed LNI 10.1.2.3@tcp [ 1060.068320] LNetError: 53178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1060.088446] LNet: Removed LNI 192.168.201.5@tcp [ 1060.115416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1060.180300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1060.197024] LNet: Added LNI 192.168.201.5@tcp [8/256/0/182] [ 1060.200142] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1060.201952] LNet: 53186:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1060.270894] LNet: Removed LNI 10.1.2.3@tcp [ 1060.279996] LNet: Removed LNI 192.168.201.5@tcp [ 1060.800957] Key type .llcrypt unregistered [ 1060.805917] Key type ._llcrypt unregistered [ 1073.110992] Key type ._llcrypt registered [ 1073.122063] Key type .llcrypt registered [ 1074.518397] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 21:03:05 (1788483785) [ 1076.563560] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1078.598324] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 21:03:08 (1788483788) [ 1079.169908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1079.193748] alg: No test for adler32 (adler32-zlib) [ 1080.198859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1080.304986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1080.343210] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 1081.404701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.5 [ 1081.424352] LNet: 54448:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1081.439569] LNet: Removed LNI 192.168.201.5@tcp [ 1087.998457] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 21:03:18 (1788483798) [ 1088.301377] LNet: 55018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1088.308585] LNetError: 55018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1088.323480] LNet: Removed LNI 192.168.201.5@tcp [ 1089.052562] Key type .llcrypt unregistered [ 1089.058382] Key type ._llcrypt unregistered [ 1096.000606] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1096.191925] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1098.919314] Key type ._llcrypt registered [ 1098.922806] Key type .llcrypt registered [ 1099.126291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1099.139571] alg: No test for adler32 (adler32-zlib) [ 1100.688292] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1111.485039] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 21:03:40 (1788483820) [ 1111.648297] LNet: 55920:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1111.664323] LNetError: 55920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1111.687593] LNet: Removed LNI 10.1.2.3@tcp [ 1112.331490] Key type .llcrypt unregistered [ 1112.333715] Key type ._llcrypt unregistered [ 1119.844515] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1120.123360] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1120.129548] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1120.133791] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1121.705938] Key type ._llcrypt registered [ 1121.708123] Key type .llcrypt registered [ 1121.905075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1121.919625] alg: No test for adler32 (adler32-zlib) [ 1122.926251] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1131.190363] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 21:04:01 (1788483841) [ 1133.059260] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1135.416648] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 21:04:05 (1788483845) [ 1135.562909] LNet: 57096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1135.571921] LNetError: 57096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1135.595182] LNet: Removed LNI 10.1.2.3@tcp [ 1136.666706] Key type .llcrypt unregistered [ 1136.671075] Key type ._llcrypt unregistered [ 1144.807606] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1145.093487] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1145.096977] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1145.101413] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1146.736803] Key type ._llcrypt registered [ 1146.744078] Key type .llcrypt registered [ 1147.062606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1147.108650] alg: No test for adler32 (adler32-zlib) [ 1148.079876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1148.142688] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1156.342802] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 21:04:27 (1788483867) [ 1156.931461] LNet: 58323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1156.937633] LNetError: 58323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1156.968181] LNet: Removed LNI 10.1.2.3@tcp [ 1157.622959] Key type .llcrypt unregistered [ 1157.629045] Key type ._llcrypt unregistered [ 1164.499802] Key type ._llcrypt registered [ 1164.501319] Key type .llcrypt registered [ 1164.774688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1164.799435] alg: No test for adler32 (adler32-zlib) [ 1165.831474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1165.855744] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 1174.123993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.105@tcp [ 1174.991932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp -m GET -e local_interrupt -r 1 [ 1175.044297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp -m GET -e local_interrupt -r 1 [ 1175.083499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.105@tcp [ 1175.112078] LNet: There was an unexpected network error while writing to 192.168.201.105: rc = -22 [ 1175.525759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1175.582386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1175.615830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1175.648820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1175.695691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1175.733557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1175.784308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1175.858790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1175.895595] LNet: 58759:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1175.916274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1175.952234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1175.986611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1176.019364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1176.052326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1176.098282] LNet: 1 local NIs in recovery (showing 1): 192.168.201.5@tcp [ 1176.099652] LNet: There was an unexpected network error while writing to 192.168.201.5: rc = -22 [ 1176.103207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1176.162317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1176.826248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp -m GET -e local_dropped -r 1 [ 1176.856632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp -m GET -e local_dropped -r 1 [ 1176.912302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.105@tcp [ 1177.196663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1177.226171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1177.260986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1177.302576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1177.336790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1177.370823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1177.423611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1177.465136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1177.483792] LNet: 58868:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1177.508557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1177.554707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1177.603381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1177.659031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1177.714795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1177.759788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1177.795198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1178.369701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp -m GET -e local_aborted -r 1 [ 1178.410671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp -m GET -e local_aborted -r 1 [ 1178.480933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.105@tcp [ 1178.500777] LNet: There was an unexpected network error while writing to 192.168.201.105: rc = -22 [ 1178.505160] LNet: Skipped 1 previous similar message [ 1178.874727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1178.928264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1178.972603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1179.044089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1179.110855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1179.183624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1179.230992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1179.273439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1179.295927] LNet: 58977:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1179.316513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1179.367905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1179.403848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1179.493470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1179.528926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1179.571612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1179.608054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1180.257670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp -m GET -e local_no_route -r 1 [ 1180.286813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp -m GET -e local_no_route -r 1 [ 1180.317505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.105@tcp [ 1180.584249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1180.613938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1180.644164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1180.671659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1180.698442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1180.728401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1180.756154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1180.789618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1180.824602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1180.896996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1180.958348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1181.005355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1181.043702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1181.111769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1181.174445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1181.778459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp -m GET -e local_timeout -r 1 [ 1181.821913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp -m GET -e local_timeout -r 1 [ 1181.857576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.105@tcp [ 1181.875965] LNet: There was an unexpected network error while writing to 192.168.201.105: rc = -22 [ 1181.882955] LNet: Skipped 1 previous similar message [ 1182.176293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1182.204827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1182.248905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1182.277362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1182.311692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1182.340608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1182.397720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1182.442698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1182.466857] LNet: 59196:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1182.483383] LNet: 59196:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1182.522409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1182.569563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1182.607370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1182.660820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1182.679666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1182.713486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1182.738684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1183.271846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp -m GET -e local_error -r 1 [ 1183.307149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp -m GET -e local_error -r 1 [ 1183.329746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.105@tcp [ 1183.542258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1183.585954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1183.611901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1183.640553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1183.664504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1183.691585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1183.743069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1183.795986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1183.836524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1183.881477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1183.915706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1183.953629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1184.002970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1184.054567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1184.101118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1184.374882] LNet: 59346:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1184.380368] LNetError: 59346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1185.467590] LNet: Removed LNI 192.168.201.5@tcp [ 1185.886400] Key type .llcrypt unregistered [ 1185.891208] Key type ._llcrypt unregistered [ 1206.698684] Key type ._llcrypt registered [ 1206.699936] Key type .llcrypt registered [ 1207.852812] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 21:05:18 (1788483918) [ 1208.750340] Key type .llcrypt unregistered [ 1208.752805] Key type ._llcrypt unregistered [ 1215.946954] Key type ._llcrypt registered [ 1215.949835] Key type .llcrypt registered [ 1216.206584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1216.242249] alg: No test for adler32 (adler32-zlib) [ 1217.292673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1217.318358] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 1226.474943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.105@tcp [ 1240.165438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1240.241127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1240.307505] LNet: Added LNI 192.168.201.5@tcp1 [8/256/0/180] [ 1242.080695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp -m GET -e local_interrupt -r 1 [ 1242.127576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp1 -m GET -e local_interrupt -r 1 [ 1242.184910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp -m GET -e local_interrupt -r 1 [ 1242.252103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp1 -m GET -e local_interrupt -r 1 [ 1242.316496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.105@tcp -m GET -e local_interrupt -r 1 [ 1242.397762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.105@tcp1 -m GET -e local_interrupt -r 1 [ 1242.465977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.5@tcp -m GET -e local_interrupt -r 1 [ 1242.535427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp1 -d 192.168.201.5@tcp1 -m GET -e local_interrupt -r 1 [ 1242.633820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.105@tcp [ 1242.677710] LNet: There was an unexpected network error while writing to 192.168.201.105: rc = -22 [ 1242.690675] LNet: 1 local NIs in recovery (showing 1): 192.168.201.5@tcp [ 1242.991531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1243.060692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1243.121189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1243.187144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1243.257276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1243.332790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1243.363667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1243.416297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1243.462195] LNet: 60813:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1243.486811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1243.536760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1243.595655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1243.644605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1243.694807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1243.753452] LNet: There was an unexpected network error while writing to 192.168.201.5: rc = -22 [ 1243.767346] LNet: Skipped 2 previous similar messages [ 1243.788243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1243.866230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1244.709506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp -m GET -e local_dropped -r 1 [ 1244.781304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp1 -m GET -e local_dropped -r 1 [ 1244.875709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp -m GET -e local_dropped -r 1 [ 1244.945698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp1 -m GET -e local_dropped -r 1 [ 1245.025679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp1 -d 192.168.201.105@tcp -m GET -e local_dropped -r 1 [ 1245.080955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp1 -d 192.168.201.105@tcp1 -m GET -e local_dropped -r 1 [ 1245.169833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.5@tcp -m GET -e local_dropped -r 1 [ 1245.222661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.5@tcp1 -m GET -e local_dropped -r 1 [ 1245.292665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.105@tcp [ 1245.331089] LNet: There was an unexpected network error while writing to 192.168.201.105: rc = -22 [ 1245.699386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1245.746710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1245.808989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1245.879549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1245.923629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1245.972603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1246.023070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1246.083487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1246.117641] LNet: 60934:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1246.142819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1246.209083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1246.261915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1246.302955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1246.352651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1246.415201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1246.483556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1247.277820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp -m GET -e local_aborted -r 1 [ 1247.345813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp1 -m GET -e local_aborted -r 1 [ 1247.390408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp -m GET -e local_aborted -r 1 [ 1247.444699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp1 -m GET -e local_aborted -r 1 [ 1247.500615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.105@tcp -m GET -e local_aborted -r 1 [ 1247.538401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp1 -d 192.168.201.105@tcp1 -m GET -e local_aborted -r 1 [ 1247.602986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp1 -d 192.168.201.5@tcp -m GET -e local_aborted -r 1 [ 1247.644778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.5@tcp1 -m GET -e local_aborted -r 1 [ 1247.705585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.105@tcp [ 1247.724305] LNet: There was an unexpected network error while writing to 192.168.201.105: rc = -22 [ 1247.728698] LNet: Skipped 2 previous similar messages [ 1248.034750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1248.075809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1248.103462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1248.138333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1248.175701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1248.204335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1248.241744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1248.270630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1248.290794] LNet: 61056:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1248.315461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1248.352503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1248.382264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1248.448938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1248.534612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1248.592859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1248.628230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1249.248870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp -m GET -e local_no_route -r 1 [ 1249.280561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp1 -m GET -e local_no_route -r 1 [ 1249.326440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp -m GET -e local_no_route -r 1 [ 1249.372273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp1 -m GET -e local_no_route -r 1 [ 1249.414545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.105@tcp -m GET -e local_no_route -r 1 [ 1249.472334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.105@tcp1 -m GET -e local_no_route -r 1 [ 1249.544515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.5@tcp -m GET -e local_no_route -r 1 [ 1249.642234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.5@tcp1 -m GET -e local_no_route -r 1 [ 1249.709814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.105@tcp [ 1250.137702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1250.166627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1250.200523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1250.232735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1250.269689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1250.298247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1250.320992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1250.342877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1250.355453] LNet: 61177:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1250.368708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1250.391752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1250.415268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1250.451306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1250.510659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1250.566968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1250.629660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1251.395602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp -m GET -e local_timeout -r 1 [ 1251.486958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp1 -m GET -e local_timeout -r 1 [ 1251.612472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp -m GET -e local_timeout -r 1 [ 1251.704520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp1 -m GET -e local_timeout -r 1 [ 1251.779898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp1 -d 192.168.201.105@tcp -m GET -e local_timeout -r 1 [ 1251.868525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp1 -d 192.168.201.105@tcp1 -m GET -e local_timeout -r 1 [ 1251.963734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp1 -d 192.168.201.5@tcp -m GET -e local_timeout -r 1 [ 1252.025992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp1 -d 192.168.201.5@tcp1 -m GET -e local_timeout -r 1 [ 1252.085969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.105@tcp [ 1252.113409] LNet: There was an unexpected network error while writing to 192.168.201.105: rc = -22 [ 1252.129079] LNet: Skipped 5 previous similar messages [ 1252.438910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1252.480440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1252.524405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1252.580324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1252.620494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1252.662995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1252.708554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1252.764564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1252.808820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1252.854797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1252.901259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1252.944415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1252.984411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1253.020495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1253.088920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1253.695173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp -m GET -e local_error -r 1 [ 1253.752215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp1 -m GET -e local_error -r 1 [ 1253.795565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp -m GET -e local_error -r 1 [ 1253.846280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp1 -m GET -e local_error -r 1 [ 1253.873864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.105@tcp -m GET -e local_error -r 1 [ 1253.930532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.105@tcp1 -m GET -e local_error -r 1 [ 1253.974404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.5@tcp -m GET -e local_error -r 1 [ 1254.013724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.5@tcp1 -m GET -e local_error -r 1 [ 1254.053646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.105@tcp [ 1254.325806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1254.365085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1254.451687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1254.507637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1254.542607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1254.600582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1254.681660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1254.733605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1254.759904] LNet: 61420:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1254.765808] LNet: 61420:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1254.806755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1254.875762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1254.948096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1255.022458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1255.082235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1255.132282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1255.188355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1263.411186] LNet: 61497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1263.437324] LNetError: 61497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1263.451092] LNet: Removed LNI 192.168.201.5@tcp [ 1264.486971] LNet: Removed LNI 192.168.201.5@tcp1 [ 1265.014121] Key type .llcrypt unregistered [ 1265.016308] Key type ._llcrypt unregistered [ 1284.829292] Key type ._llcrypt registered [ 1284.836254] Key type .llcrypt registered [ 1286.135447] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 21:06:36 (1788483996) [ 1287.210795] Key type .llcrypt unregistered [ 1287.213820] Key type ._llcrypt unregistered [ 1294.573557] Key type ._llcrypt registered [ 1294.589642] Key type .llcrypt registered [ 1295.055769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1295.084722] alg: No test for adler32 (adler32-zlib) [ 1296.146922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1296.191885] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 1304.445925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.105@tcp [ 1305.242681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp -m GET -e remote_dropped -r 1 [ 1305.265880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp -m GET -e remote_dropped -r 1 [ 1305.294629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.105@tcp [ 1305.328968] LNet: There was an unexpected network error while writing to 192.168.201.105: rc = -22 [ 1305.602417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1305.651851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1305.679447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1305.715353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1305.750352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1305.780303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1305.810503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1305.843346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1305.857178] LNet: 62743:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1305.884428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1305.926411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1305.987284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1306.033603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1306.067468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1306.101469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1306.139144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1306.638340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp -m GET -e remote_error -r 1 [ 1306.678408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp -m GET -e remote_error -r 1 [ 1306.701376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.105@tcp [ 1306.711300] LNet: There was an unexpected network error while writing to 192.168.201.105: rc = -22 [ 1306.999377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1307.045895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1307.072808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1307.114292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1307.161596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1307.180675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1307.206865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1307.251638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1307.274434] LNet: 62852:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1307.297324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1307.329308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1307.379844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1307.443643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1307.508807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1307.564311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1307.637378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1308.143719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp -m GET -e remote_timeout -r 1 [ 1308.176424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp -m GET -e remote_timeout -r 1 [ 1308.219404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.105@tcp [ 1308.251469] LNet: There was an unexpected network error while writing to 192.168.201.105: rc = -22 [ 1308.524405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1308.569578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1308.615807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1308.649875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1308.692546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1308.747987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1308.797133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1308.842332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1308.872246] LNet: 62962:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1308.894962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1308.929825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1308.969808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1309.015728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1309.051685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1309.115946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1309.154076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1309.421366] LNet: 63003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1309.432648] LNetError: 63003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1309.458307] LNet: Removed LNI 192.168.201.5@tcp [ 1309.906163] Key type .llcrypt unregistered [ 1309.913553] Key type ._llcrypt unregistered [ 1328.889235] Key type ._llcrypt registered [ 1328.893132] Key type .llcrypt registered [ 1330.009783] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 21:07:20 (1788484040) [ 1331.082303] Key type .llcrypt unregistered [ 1331.085634] Key type ._llcrypt unregistered [ 1339.060787] Key type ._llcrypt registered [ 1339.083487] Key type .llcrypt registered [ 1339.655139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1339.704638] alg: No test for adler32 (adler32-zlib) [ 1340.671785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1340.731294] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 1352.894375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.105@tcp [ 1366.765186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1366.817773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1366.853375] LNet: Added LNI 192.168.201.5@tcp1 [8/256/0/180] [ 1368.090654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp -m GET -e remote_dropped -r 1 [ 1368.149525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp1 -m GET -e remote_dropped -r 1 [ 1368.194686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp -m GET -e remote_dropped -r 1 [ 1368.242693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp1 -m GET -e remote_dropped -r 1 [ 1368.313913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp1 -d 192.168.201.105@tcp -m GET -e remote_dropped -r 1 [ 1368.406209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.105@tcp1 -m GET -e remote_dropped -r 1 [ 1368.517616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp1 -d 192.168.201.5@tcp -m GET -e remote_dropped -r 1 [ 1368.603754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp1 -d 192.168.201.5@tcp1 -m GET -e remote_dropped -r 1 [ 1368.663487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.105@tcp [ 1368.726377] LNet: There was an unexpected network error while writing to 192.168.201.105: rc = -22 [ 1368.751179] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.105@tcp [ 1369.149337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1369.185958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1369.225197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1369.266438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1369.293572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1369.346270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1369.391084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1369.437953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1369.483201] LNet: 64469:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1369.525323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1369.553219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1369.587784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1369.619771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1369.681578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1369.728590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1369.776487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1369.839611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1371.009436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp -m GET -e remote_error -r 1 [ 1371.056867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp1 -m GET -e remote_error -r 1 [ 1371.085195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp -m GET -e remote_error -r 1 [ 1371.119746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp1 -m GET -e remote_error -r 1 [ 1371.168443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp1 -d 192.168.201.105@tcp -m GET -e remote_error -r 1 [ 1371.212800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.105@tcp1 -m GET -e remote_error -r 1 [ 1371.250737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.5@tcp -m GET -e remote_error -r 1 [ 1371.282770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp1 -d 192.168.201.5@tcp1 -m GET -e remote_error -r 1 [ 1371.369283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.105@tcp [ 1371.399061] LNet: There was an unexpected network error while writing to 192.168.201.105: rc = -22 [ 1371.404659] LNet: Skipped 2 previous similar messages [ 1371.720368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1371.748551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1371.774353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1371.805920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1371.833424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1371.855494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1371.881292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1371.907846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1371.924841] LNet: 64592:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1371.971477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1372.014347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1372.065228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1372.087114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1372.113990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1372.149764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1372.191823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1372.233670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1373.021720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp -m GET -e remote_timeout -r 1 [ 1373.075227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp1 -m GET -e remote_timeout -r 1 [ 1373.131584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp -m GET -e remote_timeout -r 1 [ 1373.213641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp1 -m GET -e remote_timeout -r 1 [ 1373.258469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.105@tcp -m GET -e remote_timeout -r 1 [ 1373.291338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.105@tcp1 -m GET -e remote_timeout -r 1 [ 1373.329218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp1 -d 192.168.201.5@tcp -m GET -e remote_timeout -r 1 [ 1373.361669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.5@tcp1 -m GET -e remote_timeout -r 1 [ 1373.394451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.105@tcp [ 1373.412549] LNet: There was an unexpected network error while writing to 192.168.201.105: rc = -22 [ 1373.741549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1373.784505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1373.833911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1373.887852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1373.958681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1373.995854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1374.028200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1374.077439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1374.116996] LNet: 64715:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1374.156993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1374.227413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1374.295341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1374.330864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1374.369830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1374.396986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1374.419657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1374.445698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1381.997533] LNet: 64794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1382.003684] LNetError: 64794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1382.012772] LNet: Removed LNI 192.168.201.5@tcp [ 1382.027209] LNet: Removed LNI 192.168.201.5@tcp1 [ 1382.732738] Key type .llcrypt unregistered [ 1382.734743] Key type ._llcrypt unregistered [ 1407.246244] Key type ._llcrypt registered [ 1407.253474] Key type .llcrypt registered [ 1408.314535] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 21:08:39 (1788484119) [ 1410.098721] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1412.246251] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 21:08:42 (1788484122) [ 1413.729557] Key type .llcrypt unregistered [ 1413.732090] Key type ._llcrypt unregistered [ 1420.815569] Key type ._llcrypt registered [ 1420.819669] Key type .llcrypt registered [ 1421.059040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1421.076833] alg: No test for adler32 (adler32-zlib) [ 1422.053551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1422.124642] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 1429.820598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.105@tcp [ 1430.681492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp -m GET -e network_timeout -r 1 [ 1430.740427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp -m GET -e network_timeout -r 1 [ 1430.790321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.105@tcp [ 1430.806930] LNet: There was an unexpected network error while writing to 192.168.201.105: rc = -22 [ 1431.085166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1431.122875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1431.162353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1431.195621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1431.224785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1431.255411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1431.296369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1431.326172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1431.328579] LNet: 1 local NIs in recovery (showing 1): 192.168.201.5@tcp [ 1431.329907] LNet: There was an unexpected network error while writing to 192.168.201.5: rc = -22 [ 1431.358454] LNet: 66337:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1431.381449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1431.414530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1431.441506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1431.467647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1431.502740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1431.546690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1431.838777] LNet: 66376:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1431.847466] LNetError: 66376:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1432.889349] LNet: Removed LNI 192.168.201.5@tcp [ 1433.579523] Key type .llcrypt unregistered [ 1433.584363] Key type ._llcrypt unregistered [ 1453.965357] Key type ._llcrypt registered [ 1453.978524] Key type .llcrypt registered [ 1454.455824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1454.520426] alg: No test for adler32 (adler32-zlib) [ 1455.543487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1455.573228] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 1466.055766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.105@tcp [ 1477.984373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1478.089418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1478.103343] LNet: Added LNI 192.168.201.5@tcp1 [8/256/0/180] [ 1479.638228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp -m GET -e network_timeout -r 1 [ 1479.712605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp1 -m GET -e network_timeout -r 1 [ 1479.764539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp -m GET -e network_timeout -r 1 [ 1479.796759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp1 -m GET -e network_timeout -r 1 [ 1479.825599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.105@tcp -m GET -e network_timeout -r 1 [ 1479.879238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp1 -d 192.168.201.105@tcp1 -m GET -e network_timeout -r 1 [ 1479.942355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp1 -d 192.168.201.5@tcp -m GET -e network_timeout -r 1 [ 1479.974218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.5@tcp1 -m GET -e network_timeout -r 1 [ 1480.014815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.105@tcp [ 1480.055714] LNet: There was an unexpected network error while writing to 192.168.201.105: rc = -22 [ 1480.163424] LNet: 1 local NIs in recovery (showing 1): 192.168.201.5@tcp [ 1480.179524] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.105@tcp [ 1480.357605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1480.390766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1480.432786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1480.485964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1480.535547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1480.607822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1480.655969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1480.738636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1480.758935] LNet: 67253:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1480.775961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1480.802354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1480.837468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1480.894507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1480.945393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1480.989755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1485.791548] LNet: There was an unexpected network error while writing to 192.168.201.105: rc = -22 [ 1486.222031] LNet: 67327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1486.232496] LNetError: 67327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1486.245852] LNet: Removed LNI 192.168.201.5@tcp [ 1486.265384] LNet: Removed LNI 192.168.201.5@tcp1 [ 1486.849755] Key type .llcrypt unregistered [ 1486.852300] Key type ._llcrypt unregistered [ 1505.111157] Key type ._llcrypt registered [ 1505.114469] Key type .llcrypt registered [ 1506.590663] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 21:10:17 (1788484217) [ 1507.167406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1507.196348] alg: No test for adler32 (adler32-zlib) [ 1508.211441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1508.313478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1508.365388] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 1509.367406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1509.395879] LNet: Added LNI 192.168.201.5@tcp1 [8/256/0/180] [ 1510.358758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.5@tcp [ 1510.466633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1510.554560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1510.640981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1510.724717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1510.791767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1510.847809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.5@tcp [ 1510.870299] LNetError: 68427:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.5@tcp to recovery [ 1511.328441] LNet: 1 local NIs in recovery (showing 1): 192.168.201.5@tcp [ 1511.329929] LNet: There was an unexpected network error while writing to 192.168.201.5: rc = -22 [ 1513.379131] LNet: There was an unexpected network error while writing to 192.168.201.5: rc = -22 [ 1514.693052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1517.472398] LNet: There was an unexpected network error while writing to 192.168.201.5: rc = -22 [ 1518.432544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1518.472393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1518.525946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1518.554476] LNet: 68535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1518.563347] LNetError: 68535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1518.578930] LNet: Removed LNI 192.168.201.5@tcp [ 1518.593871] LNet: Removed LNI 192.168.201.5@tcp1 [ 1518.614981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1518.686237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1518.713588] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 1519.479186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1519.509893] LNet: Added LNI 192.168.201.5@tcp1 [8/256/0/180] [ 1520.508299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.5@tcp [ 1520.539826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1520.595579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1520.643883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1520.679338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1520.712060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1520.746394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1520.806364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.5@tcp [ 1520.845938] LNetError: 68807:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.5@tcp to recovery [ 1521.761089] LNet: There was an unexpected network error while writing to 192.168.201.5: rc = -22 [ 1524.683134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1527.905372] LNet: There was an unexpected network error while writing to 192.168.201.5: rc = -22 [ 1527.911916] LNet: Skipped 1 previous similar message [ 1532.944395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1533.008288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1533.099756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1542.300872] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 21:10:52 (1788484252) [ 1542.438230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1542.473032] LNet: 69519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1542.480100] LNetError: 69519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1542.487592] LNetError: 69519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1542.497610] LNet: Removed LNI 192.168.201.5@tcp [ 1542.531946] LNet: Removed LNI 192.168.201.5@tcp1 [ 1542.623721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1542.698877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1542.736497] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 1543.459742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1543.490506] LNet: Added LNI 192.168.201.5@tcp1 [8/256/0/180] [ 1544.547658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.5@tcp [ 1544.611654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1544.641814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1544.670678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1544.694493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1544.715138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1544.748380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.5@tcp --health 0 [ 1544.892580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1545.760898] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.5@tcp [ 1545.761243] LNet: There was an unexpected network error while writing to 192.168.201.5: rc = -22 [ 1545.768107] LNet: Skipped 1 previous similar message [ 1559.360194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1559.489933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1559.514438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1559.540667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.5@tcp --health 500 [ 1559.671826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1565.622442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1565.748501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1565.793157] LNet: 70000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1565.809408] LNetError: 70000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1565.820641] LNetError: 70000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1565.831309] LNet: Removed LNI 192.168.201.5@tcp [ 1565.852281] LNet: Removed LNI 192.168.201.5@tcp1 [ 1565.898192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1566.059429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1566.147392] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 1567.655812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1567.684048] LNet: Added LNI 192.168.201.5@tcp1 [8/256/0/180] [ 1569.977538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.5@tcp [ 1570.109763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1570.318082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1570.397314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1570.449558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1570.553791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1570.642187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1570.761300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.5@tcp --health 0 [ 1571.104792] LNet: There was an unexpected network error while writing to 192.168.201.5: rc = -22 [ 1571.115945] LNet: Skipped 3 previous similar messages [ 1572.333983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1582.851410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1582.921883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1582.974794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1593.133705] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 21:11:43 (1788484303) [ 1593.479638] LNet: 71037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1593.485029] LNetError: 71037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1593.494284] LNetError: 71037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1593.506553] LNet: Removed LNI 192.168.201.5@tcp [ 1593.526110] LNet: Removed LNI 192.168.201.5@tcp1 [ 1594.299928] Key type .llcrypt unregistered [ 1594.301714] Key type ._llcrypt unregistered [ 1603.868939] Key type ._llcrypt registered [ 1603.876486] Key type .llcrypt registered [ 1604.157144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1604.182252] alg: No test for adler32 (adler32-zlib) [ 1605.138354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1605.162883] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 1614.927291] Lustre: DEBUG MARKER: Initial discovery [ 1615.024107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.105@tcp [ 1617.973035] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1618.076684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1618.146034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.105@tcp [ 1618.208440] LNet: There was an unexpected network error while writing to 192.168.201.105: rc = -22 [ 1618.252430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1618.323447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.105@tcp -r 1 -m GET -l 3 [ 1618.420398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.105@tcp -r 1 -m GET -e local_error [ 1618.492852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.105@tcp -r 1 -m PUT -l 6 [ 1618.536880] LNet: 1 local NIs in recovery (showing 1): 192.168.201.5@tcp [ 1620.102165] Lustre: DEBUG MARKER: Force oleg105-client.virtnet to discover 192.168.201.105@tcp (in background) [ 1620.303143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.105@tcp [ 1622.197610] Lustre: DEBUG MARKER: Force oleg105-server to discover 192.168.201.5@tcp [ 1623.393105] LNet: There was an unexpected network error while writing to 192.168.201.105: rc = -22 [ 1625.274454] Lustre: DEBUG MARKER: Wait for 71800 [ 1627.064393] Lustre: DEBUG MARKER: Finished wait on 71800 [ 1627.278547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1627.328792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1627.340568] LNet: There was an unexpected network error while writing to 192.168.201.105: rc = -22 [ 1627.574580] LNet: 72249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1627.579545] LNetError: 72249:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1627.594184] LNet: Removed LNI 192.168.201.5@tcp [ 1628.016597] Key type .llcrypt unregistered [ 1628.019133] Key type ._llcrypt unregistered [ 1646.875191] Key type ._llcrypt registered [ 1646.877921] Key type .llcrypt registered [ 1648.099882] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 21:12:38 (1788484358) [ 1648.625400] Key type .llcrypt unregistered [ 1648.628182] Key type ._llcrypt unregistered [ 1654.852965] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1655.025613] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1655.035734] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1655.039196] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1657.529935] Key type ._llcrypt registered [ 1657.531158] Key type .llcrypt registered [ 1657.640593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1657.657221] alg: No test for adler32 (adler32-zlib) [ 1658.533076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1658.573886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1658.591527] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 1659.327197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1659.342336] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1669.071421] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 21:12:59 (1788484379) [ 1669.871194] LNet: 74323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1669.877504] LNetError: 74323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1669.885599] LNet: Removed LNI 10.1.2.3@tcp [ 1669.921130] LNet: Removed LNI 192.168.201.5@tcp [ 1671.195485] Key type .llcrypt unregistered [ 1671.197338] Key type ._llcrypt unregistered [ 1680.995460] Key type ._llcrypt registered [ 1680.997407] Key type .llcrypt registered [ 1681.446957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1681.496860] alg: No test for adler32 (adler32-zlib) [ 1682.648393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1682.792151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1682.883735] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 1684.553466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1684.613361] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1686.264935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.5@tcp [ 1687.328223] LNetError: Unexpected error -22 connecting to 192.168.201.5@tcp at host 192.168.201.5:988 [ 1695.131939] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 21:13:25 (1788484405) [ 1695.871711] LNet: 75454:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1695.879960] LNetError: 75454:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1695.894906] LNet: Removed LNI 10.1.2.3@tcp [ 1695.920520] LNet: Removed LNI 192.168.201.5@tcp [ 1696.883505] Key type .llcrypt unregistered [ 1696.891819] Key type ._llcrypt unregistered [ 1706.078403] Key type ._llcrypt registered [ 1706.081083] Key type .llcrypt registered [ 1706.472948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1706.527839] alg: No test for adler32 (adler32-zlib) [ 1707.502345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1707.595938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1707.612953] LNet: Added LNI 192.168.201.5@tcp1 [8/256/0/180] [ 1708.456692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1708.474469] LNet: Added LNI 192.168.201.5@tcp2 [8/256/0/180] [ 1709.834799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.5@tcp1 --nid 192.168.201.5@tcp2 [ 1716.841739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.5@tcp1 192.168.201.5@tcp2 [ 1716.869307] LNetError: 76288:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.5@tcp1 to 192.168.201.5@tcp2 - Route table may be misconfigured [ 1716.900680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.5@tcp2 192.168.201.5@tcp1 [ 1725.710655] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 21:13:56 (1788484436) [ 1725.912605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1725.952848] LNet: 76862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1725.971562] LNetError: 76862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1725.979270] LNet: Removed LNI 192.168.201.5@tcp1 [ 1727.018495] LNet: Removed LNI 192.168.201.5@tcp2 [ 1727.072738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1727.151475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1727.192875] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 1727.960554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1727.971099] LNet: Added LNI 192.168.201.5@tcp1 [8/256/0/180] [ 1729.022963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.5@tcp [ 1729.088282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.5@tcp 192.168.201.5@tcp [ 1729.159191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.5@tcp1 192.168.201.5@tcp1 [ 1729.239665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.5@tcp -d 192.168.201.5@tcp -e network_timeout [ 1729.316551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.5@tcp -d 192.168.201.5@tcp1 -e network_timeout [ 1729.365735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.5@tcp1 -d 192.168.201.5@tcp -e network_timeout [ 1729.404526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.5@tcp1 -d 192.168.201.5@tcp1 -e network_timeout [ 1729.455478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.5@tcp [ 1729.484610] LNet: There was an unexpected network error while writing to 192.168.201.5: rc = -22 [ 1729.519736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1729.633438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1729.768358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1730.208347] LNet: 1 local NIs in recovery (showing 1): 192.168.201.5@tcp [ 1739.042148] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 21:14:08 (1788484448) [ 1739.148954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1739.190095] LNet: 77719:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1739.211492] LNetError: 77719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1739.229266] LNetError: 77719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1739.252708] LNet: Removed LNI 192.168.201.5@tcp [ 1740.334520] LNet: Removed LNI 192.168.201.5@tcp1 [ 1740.399236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1740.671962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1741.353430] LNet: 77763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1741.740652] Key type .llcrypt unregistered [ 1741.743909] Key type ._llcrypt unregistered [ 1758.348487] Key type ._llcrypt registered [ 1758.351580] Key type .llcrypt registered [ 1759.976121] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 21:14:30 (1788484470) [ 1760.404463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1760.420653] alg: No test for adler32 (adler32-zlib) [ 1761.431986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1763.559502] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1765.849593] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 21:14:36 (1788484476) [ 1766.000782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1766.070971] LNet: 78854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1766.119991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1766.316888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1766.341052] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 1767.606876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1767.636500] LNet: Added LNI 192.168.201.5@tcp1 [8/256/0/180] [ 1769.249017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.5@tcp [ 1769.285505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.5@tcp1 [ 1769.341624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.5@tcp1 [ 1778.817513] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 21:14:48 (1788484488) [ 1781.605631] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg105-server [ 1783.225805] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 21:14:54 (1788484494) [ 1785.687908] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg105-server [ 1788.029421] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 21:14:58 (1788484498) [ 1790.739650] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg105-server [ 1792.778914] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 21:15:03 (1788484503) [ 1795.039446] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg105-server [ 1796.572313] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 21:15:07 (1788484507) [ 1798.170477] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg105-server [ 1801.461269] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 21:15:11 (1788484511) [ 1805.238994] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg105-server [ 1808.176435] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 21:15:17 (1788484517) [ 1810.934979] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg105-server [ 1814.668555] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 21:15:23 (1788484523) [ 1817.012834] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg105-server [ 1819.479934] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 21:15:29 (1788484529) [ 1821.021985] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg105-server [ 1822.918409] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 21:15:33 (1788484533) [ 1823.562727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1823.592538] LNet: 82740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1823.601845] LNetError: 82740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1823.608616] LNet: Removed LNI 192.168.201.5@tcp [ 1824.680041] LNet: Removed LNI 192.168.201.5@tcp1 [ 1824.739929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1824.811812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1824.884703] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 1825.971240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1826.007469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.5@tcp [ 1826.192929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1826.209623] LNet: 82888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1826.214673] LNetError: 82888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1826.225788] LNetError: 82888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1826.251701] LNet: Removed LNI 192.168.201.5@tcp [ 1826.266905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1826.344921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1826.397839] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 1827.354125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.5@tcp [ 1828.639901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1828.665311] LNet: 83042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1828.669993] LNetError: 83042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1829.737079] LNet: Removed LNI 192.168.201.5@tcp [ 1829.807941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1829.974084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1830.026376] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 1831.040150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1831.126221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.5@tcp [ 1831.338868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1831.367579] LNetError: 83190:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1831.387841] LNet: Removed LNI 192.168.201.5@tcp [ 1831.425636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1831.500477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1831.546808] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 1832.272949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.5@tcp [ 1832.428840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1833.511282] LNet: Removed LNI 192.168.201.5@tcp [ 1833.581753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1833.730054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1833.786661] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 1834.891923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1834.952426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.5@tcp [ 1835.185694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1835.250281] LNet: 83483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1835.260454] LNet: 83483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1836.333792] LNet: Removed LNI 192.168.201.5@tcp [ 1836.363180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1836.432941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1836.473959] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 1837.145814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.5@tcp [ 1837.385692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1837.435907] LNetError: 83628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1837.440148] LNetError: 83628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1837.474447] LNet: Removed LNI 192.168.201.5@tcp [ 1837.496273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1837.564350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1837.634780] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 1838.705251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1838.775272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.5@tcp [ 1840.171698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1841.255617] LNet: Removed LNI 192.168.201.5@tcp [ 1841.279564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1841.334931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1841.361599] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 1842.323432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.5@tcp [ 1842.621878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1843.710618] LNet: Removed LNI 192.168.201.5@tcp [ 1843.795045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1843.938974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1844.014790] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 1844.892711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1844.978123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.5@tcp [ 1845.314711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1845.344933] LNet: 84078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1845.350938] LNet: 84078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1846.385729] LNet: Removed LNI 192.168.201.5@tcp [ 1846.435988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1846.544846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1846.588671] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 1848.144427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.5@tcp [ 1849.558817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1849.586531] LNetError: 84232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1849.590992] LNetError: 84232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1850.661801] LNet: Removed LNI 192.168.201.5@tcp [ 1850.697433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1850.787505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1850.816266] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 1851.736379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1851.840615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.5@tcp [ 1851.992456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1853.030880] LNet: Removed LNI 192.168.201.5@tcp [ 1853.067864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1853.179846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1853.219490] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 1854.024441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.5@tcp [ 1854.294375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1855.404997] LNet: Removed LNI 192.168.201.5@tcp [ 1855.452033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1855.519594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1855.547720] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 1856.511578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1856.622342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1856.672547] LNet: Removed LNI 192.168.201.5@tcp [ 1856.705226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1856.789492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1856.817387] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 1857.906841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1858.030512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1858.091897] LNet: Removed LNI 192.168.201.5@tcp [ 1858.124882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1858.238544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1858.372396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1858.459270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1858.549095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1858.680421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1858.751488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1858.882854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1859.056868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1859.138385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1859.215075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1859.243872] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 1860.208817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1860.239055] LNet: Removed LNI 192.168.201.5@tcp [ 1860.270915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1860.402775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1870.349784] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 21:16:20 (1788484580) [ 1870.451980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1870.478308] LNet: 85540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1870.486406] LNet: 85540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1870.540714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1870.685761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1870.730768] LNet: Added LNI 192.168.201.5@tcp231 [8/256/0/180] [ 1872.107853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1872.150405] LNetError: 85686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1872.155387] LNetError: 85686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1872.176361] LNet: Removed LNI 192.168.201.5@tcp231 [ 1872.210655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1872.330319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1872.375160] LNet: Added LNI 192.168.201.5@tcp231 [8/256/0/180] [ 1872.382610] LNet: 85694:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1873.081569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1873.151122] LNet: Removed LNI 192.168.201.5@tcp231 [ 1873.177492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1873.210359] LNet: Added LNI 192.168.201.5@tcp231 [8/256/0/180] [ 1874.997996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1875.046729] LNet: Removed LNI 192.168.201.5@tcp231 [ 1875.071772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1875.134844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1875.167762] LNet: Added LNI 192.168.201.5@tcp231 [8/256/0/0] [ 1875.172143] LNet: 85846:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1875.938819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1875.978050] LNet: Removed LNI 192.168.201.5@tcp231 [ 1876.003288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1876.042518] LNet: Added LNI 192.168.201.5@tcp231 [8/256/0/0] [ 1877.730774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1877.809257] LNet: Removed LNI 192.168.201.5@tcp231 [ 1877.868829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1878.047294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1878.092403] LNet: Added LNI 192.168.201.5@tcp231 [8/256/0/60] [ 1878.099907] LNet: 85998:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1878.522350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1878.587266] LNet: Removed LNI 192.168.201.5@tcp231 [ 1878.631358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1878.669208] LNet: Added LNI 192.168.201.5@tcp231 [8/256/0/60] [ 1880.715423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1880.774674] LNet: Removed LNI 192.168.201.5@tcp231 [ 1880.823362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1880.929245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1880.971637] LNet: Added LNI 192.168.201.5@tcp231 [8/256/0/180] [ 1880.974783] LNet: 86149:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1881.190574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1881.216984] LNet: Removed LNI 192.168.201.5@tcp231 [ 1881.226297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1881.237922] LNet: Added LNI 192.168.201.5@tcp231 [8/256/0/180] [ 1891.578441] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 21:16:42 (1788484602) [ 1891.806194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1891.881517] LNet: Removed LNI 192.168.201.5@tcp231 [ 1891.908816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1891.997594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1892.035292] LNet: Added LNI 192.168.201.5@tcp232 [8/256/0/180] [ 1893.159971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1893.194122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1893.229322] LNet: Removed LNI 192.168.201.5@tcp232 [ 1893.242161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1893.266949] LNet: Added LNI 192.168.201.5@tcp232 [8/256/0/180] [ 1894.093864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1894.159930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1894.224470] LNet: Removed LNI 192.168.201.5@tcp232 [ 1894.247021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1894.290474] LNet: Added LNI 192.168.201.5@tcp232 [8/256/0/180] [ 1895.492756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1895.523783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1895.555859] LNet: Removed LNI 192.168.201.5@tcp232 [ 1895.571691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1895.606283] LNet: Added LNI 192.168.201.5@tcp232 [8/256/0/180] [ 1896.543132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1896.583816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1896.608974] LNet: Removed LNI 192.168.201.5@tcp232 [ 1896.620177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1903.150389] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 21:16:53 (1788484613) [ 1903.675708] LNet: 87989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1903.682811] LNet: 87989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1904.711394] Key type .llcrypt unregistered [ 1904.713356] Key type ._llcrypt unregistered [ 1911.685245] Key type ._llcrypt registered [ 1911.687498] Key type .llcrypt registered [ 1912.064645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1912.100892] alg: No test for adler32 (adler32-zlib) [ 1913.189354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1913.232898] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 1922.671983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.105@tcp [ 1936.670240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1936.741388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1936.820392] LNet: Added LNI 192.168.201.5@tcp1 [8/256/0/180] [ 1939.151209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp -m GET -e local_interrupt -r 2 [ 1939.197973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp1 -m GET -e local_interrupt -r 2 [ 1939.245562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp -m GET -e local_interrupt -r 2 [ 1939.291867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp1 -m GET -e local_interrupt -r 2 [ 1939.407255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.105@tcp -m GET -e local_interrupt -r 2 [ 1939.476465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.105@tcp1 -m GET -e local_interrupt -r 2 [ 1939.532898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.5@tcp -m GET -e local_interrupt -r 2 [ 1939.587543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.5@tcp1 -m GET -e local_interrupt -r 2 [ 1939.640325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.5@tcp 192.168.201.105@tcp [ 1939.671578] LNet: There was an unexpected network error while writing to 192.168.201.105: rc = -22 [ 1939.679805] LNet: 1 local NIs in recovery (showing 1): 192.168.201.5@tcp [ 1940.079742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1940.127562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1940.182478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1940.214706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1940.259444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1940.315834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1940.391368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1940.435197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1940.496351] LNet: 88650:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1940.516533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1940.565350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1940.620800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1940.657477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1940.704209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1940.730994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1941.312217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp -m GET -e local_dropped -r 2 [ 1941.359939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp1 -m GET -e local_dropped -r 2 [ 1941.438941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp -m GET -e local_dropped -r 2 [ 1941.470902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp1 -m GET -e local_dropped -r 2 [ 1941.514348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp1 -d 192.168.201.105@tcp -m GET -e local_dropped -r 2 [ 1941.540938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.105@tcp1 -m GET -e local_dropped -r 2 [ 1941.589689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp1 -d 192.168.201.5@tcp -m GET -e local_dropped -r 2 [ 1941.640447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.5@tcp1 -m GET -e local_dropped -r 2 [ 1941.678888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.5@tcp 192.168.201.105@tcp [ 1941.702851] LNet: There was an unexpected network error while writing to 192.168.201.105: rc = -22 [ 1941.910493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1941.929712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1941.949918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1941.979647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1942.003809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1942.024829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1942.053760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1942.081642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1942.100298] LNet: 88770:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1942.125708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1942.158547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1942.190209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1942.220393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1942.262891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1942.322715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1942.921450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp -m GET -e local_aborted -r 2 [ 1942.972571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp1 -m GET -e local_aborted -r 2 [ 1943.013747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp -m GET -e local_aborted -r 2 [ 1943.051570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp1 -m GET -e local_aborted -r 2 [ 1943.083992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.105@tcp -m GET -e local_aborted -r 2 [ 1943.120204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.105@tcp1 -m GET -e local_aborted -r 2 [ 1943.173574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.5@tcp -m GET -e local_aborted -r 2 [ 1943.223769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.5@tcp1 -m GET -e local_aborted -r 2 [ 1943.275362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.5@tcp 192.168.201.105@tcp [ 1943.291867] LNet: There was an unexpected network error while writing to 192.168.201.105: rc = -22 [ 1943.499856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1943.525939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1943.553301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1943.580864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1943.621330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1943.654836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1943.728213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1943.758913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1943.780635] LNet: 88890:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1943.801538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1943.833927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1943.866389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1943.908612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1943.948568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1943.985474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1944.722659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp -m GET -e local_no_route -r 2 [ 1944.759519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp1 -m GET -e local_no_route -r 2 [ 1944.791764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp -m GET -e local_no_route -r 2 [ 1944.819952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp1 -m GET -e local_no_route -r 2 [ 1944.870771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.105@tcp -m GET -e local_no_route -r 2 [ 1944.910983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.105@tcp1 -m GET -e local_no_route -r 2 [ 1944.956728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp1 -d 192.168.201.5@tcp -m GET -e local_no_route -r 2 [ 1944.982537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp1 -d 192.168.201.5@tcp1 -m GET -e local_no_route -r 2 [ 1945.042944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.5@tcp 192.168.201.105@tcp [ 1945.385629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1945.428377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1945.491450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1945.521800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1945.544857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1945.560778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1945.589357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1945.620447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1945.649497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1945.679426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1945.717817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1945.737313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1945.761660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1945.815217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1946.221759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp -m GET -e local_timeout -r 2 [ 1946.327979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp1 -m GET -e local_timeout -r 2 [ 1946.383741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp -m GET -e local_timeout -r 2 [ 1946.416805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp1 -m GET -e local_timeout -r 2 [ 1946.447470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.105@tcp -m GET -e local_timeout -r 2 [ 1946.501346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.105@tcp1 -m GET -e local_timeout -r 2 [ 1946.559491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp1 -d 192.168.201.5@tcp -m GET -e local_timeout -r 2 [ 1946.606639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.5@tcp1 -m GET -e local_timeout -r 2 [ 1946.644280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.5@tcp 192.168.201.105@tcp [ 1946.671952] LNet: There was an unexpected network error while writing to 192.168.201.105: rc = -22 [ 1946.674968] LNet: Skipped 1 previous similar message [ 1946.874584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1946.891670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1946.909957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1946.933900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1946.963063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1946.986878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1947.007430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1947.025543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1947.040757] LNet: 89131:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1947.044343] LNet: 89131:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1947.058491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1947.074452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1947.099516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1947.122555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1947.138247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1947.159615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1947.519482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp -m GET -e remote_dropped -r 2 [ 1947.539299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.105@tcp1 -m GET -e remote_dropped -r 2 [ 1947.558902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp -m GET -e remote_dropped -r 2 [ 1947.581822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp -d 192.168.201.5@tcp1 -m GET -e remote_dropped -r 2 [ 1947.607192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp1 -d 192.168.201.105@tcp -m GET -e remote_dropped -r 2 [ 1947.640913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.105@tcp1 -m GET -e remote_dropped -r 2 [ 1947.735651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.5@tcp1 -d 192.168.201.5@tcp -m GET -e remote_dropped -r 2 [ 1947.768118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.5@tcp1 -d 192.168.201.5@tcp1 -m GET -e remote_dropped -r 2 [ 1947.809522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.5@tcp 192.168.201.105@tcp [ 1947.823845] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.105@tcp [ 1948.002750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1948.020434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1948.049062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1948.085227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1948.126848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1948.155985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1948.232908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1948.293340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1948.334676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1948.360626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1948.386883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1948.415312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1948.438713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1948.463855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1954.140701] LNet: 89325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1954.147512] LNetError: 89325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1954.154714] LNet: Removed LNI 192.168.201.5@tcp [ 1954.168807] LNet: Removed LNI 192.168.201.5@tcp1 [ 1954.739947] Key type .llcrypt unregistered [ 1954.744412] Key type ._llcrypt unregistered [ 1978.945653] Key type ._llcrypt registered [ 1978.953556] Key type .llcrypt registered [ 1981.236444] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 21:18:10 (1788484690) [ 1981.719679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1981.735506] alg: No test for adler32 (adler32-zlib) [ 1982.685464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1982.724743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1982.752720] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 1984.117934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1984.186439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.5@tcp [ 1984.217673] LNetError: 90304:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.5@tcp to recovery [ 1984.343323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.5@tcp [ 1984.425767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1992.255730] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 21:18:22 (1788484702) [ 1992.386562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1992.457786] LNet: 90886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1992.493430] LNetError: 90886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1992.528494] LNet: Removed LNI 192.168.201.5@tcp [ 1992.560325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1992.675531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1992.774854] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 1994.006445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1994.033860] LNet: Added LNI 192.168.201.5@tcp1 [8/256/0/180] [ 1995.329840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.5@tcp [ 1995.350408] LNetError: 91160:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.5@tcp to recovery [ 1995.681983] LNet: 1 local NIs in recovery (showing 1): 192.168.201.5@tcp [ 1995.685197] LNet: There was an unexpected network error while writing to 192.168.201.5: rc = -22 [ 1996.654660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.5@tcp [ 2003.830819] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 21:18:34 (1788484714) [ 2006.034567] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg105-server [ 2007.980494] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 21:18:38 (1788484718) [ 2010.433275] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg105-server [ 2012.540499] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 21:18:42 (1788484722) [ 2014.691643] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg105-server [ 2017.135887] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 21:18:47 (1788484727) [ 2017.286402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2017.336611] LNet: 92776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2017.355424] LNetError: 92776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2017.378317] LNet: Removed LNI 192.168.201.5@tcp [ 2017.466341] LNet: Removed LNI 192.168.201.5@tcp1 [ 2017.571420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2017.670878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2017.741601] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 2018.414549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 2018.471063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 2018.497752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2018.558658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 2018.601485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2018.658437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 2025.710261] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 21:18:56 (1788484736) [ 2026.183597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2026.219347] LNet: 93515:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2026.230910] LNetError: 93515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2026.233929] LNetError: 93515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2026.252707] LNet: Removed LNI 192.168.201.5@tcp [ 2026.295029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2026.360675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2026.391392] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 2034.338497] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 21:19:04 (1788484744) [ 2034.872588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2034.923334] LNet: 94219:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2034.933144] LNetError: 94219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2034.948076] LNet: Removed LNI 192.168.201.5@tcp [ 2034.997467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2035.096315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2035.132460] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 2042.466290] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 21:19:13 (1788484753) [ 2044.283698] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 2046.437296] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 21:19:17 (1788484757) [ 2047.403554] LNet: 95254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2047.410533] LNetError: 95254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2047.425152] LNet: Removed LNI 192.168.201.5@tcp [ 2048.255654] Key type .llcrypt unregistered [ 2048.258308] Key type ._llcrypt unregistered [ 2058.134771] Key type ._llcrypt registered [ 2058.136637] Key type .llcrypt registered [ 2058.482430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2058.520323] alg: No test for adler32 (adler32-zlib) [ 2059.482457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2059.506539] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 2069.332624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.105@tcp [ 2076.060969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.105@tcp [ 2076.585395] LNet: 95669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2076.594429] LNetError: 95669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2076.621326] LNet: Removed LNI 192.168.201.5@tcp [ 2077.262437] Key type .llcrypt unregistered [ 2077.264645] Key type ._llcrypt unregistered [ 2090.097273] Key type ._llcrypt registered [ 2090.099379] Key type .llcrypt registered [ 2091.390065] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2093.359987] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2095.944392] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 21:20:06 (1788484806) [ 2098.215144] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2101.299646] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 21:20:10 (1788484810) [ 2103.505389] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg105-server [ 2105.268350] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 21:20:15 (1788484815) [ 2107.100825] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg105-server [ 2109.472243] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 21:20:20 (1788484820) [ 2111.374921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2111.488660] alg: No test for adler32 (adler32-zlib) [ 2112.926628] Lustre: Lustre: Build Version: 2.17.57_84_g5b8c505 [ 2113.377688] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 2115.104606] Key type lgssc registered [ 2117.333644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2145.517747] Key type lgssc unregistered [ 2145.696440] LNet: 98842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2145.703557] LNetError: 98842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2145.728235] LNet: Removed LNI 192.168.201.5@tcp [ 2146.720550] Key type .llcrypt unregistered [ 2146.722409] Key type ._llcrypt unregistered [ 2169.760860] Key type ._llcrypt registered [ 2169.768367] Key type .llcrypt registered [ 2172.170697] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 21:21:21 (1788484881) [ 2173.012243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2173.055132] alg: No test for adler32 (adler32-zlib) [ 2174.081364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2174.204108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2174.314634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2174.403025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2174.441453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2183.865488] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 21:21:33 (1788484893) [ 2184.092355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2184.136652] LNet: 100250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2184.197287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2184.324642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2184.360448] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 2186.094327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2186.148674] LNet: 100400:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2186.162911] LNetError: 100400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2186.193846] LNet: Removed LNI 192.168.201.5@tcp [ 2186.224856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2186.331771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2186.401888] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 2187.321825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2187.395554] LNet: 100428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2187.418786] LNetError: 100428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2187.448429] LNet: Removed LNI 192.168.201.5@tcp [ 2187.503782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2187.638820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2187.710924] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 2199.070304] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 21:21:48 (1788484908) [ 2199.529498] LNet: 101020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2199.553224] LNetError: 101020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2199.592731] LNet: Removed LNI 192.168.201.5@tcp [ 2200.650697] Key type .llcrypt unregistered [ 2200.652389] Key type ._llcrypt unregistered [ 2208.758393] Key type ._llcrypt registered [ 2208.761884] Key type .llcrypt registered [ 2209.006824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2209.021208] alg: No test for adler32 (adler32-zlib) [ 2209.971403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2210.036873] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 2210.039747] LNet: *** cfs_fail_loc=e003, val=100*** [ 2210.680492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.5@tcp [ 2210.698490] LNet: *** cfs_fail_loc=e003, val=100*** [ 2210.701409] LNet: Skipped 2 previous similar messages [ 2210.708804] Ping reply received with mapping: [ 2210.711342] found metadata reply NI[192.168.201.5@tcp] [ 2210.713080] 05 c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2210.715381] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2210.724717] 5a 5a 5a 5a 5a 5a 5a 5a [ 2210.888932] LNet: 101415:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2210.892518] LNetError: 101415:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2211.945648] LNet: Removed LNI 192.168.201.5@tcp [ 2212.665479] Key type .llcrypt unregistered [ 2212.670458] Key type ._llcrypt unregistered [ 2225.637593] Key type ._llcrypt registered [ 2225.640532] Key type .llcrypt registered [ 2227.009614] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 21:22:17 (1788484937) [ 2227.587683] Key type .llcrypt unregistered [ 2227.588968] Key type ._llcrypt unregistered [ 2236.446922] Key type ._llcrypt registered [ 2236.452731] Key type .llcrypt registered [ 2236.990062] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2237.007418] alg: No test for adler32 (adler32-zlib) [ 2238.031642] LNetError: 102452:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2238.342535] Key type .llcrypt unregistered [ 2238.343838] Key type ._llcrypt unregistered [ 2244.347881] Key type ._llcrypt registered [ 2244.350194] Key type .llcrypt registered [ 2245.726977] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 21:22:36 (1788484956) [ 2246.326424] Key type .llcrypt unregistered [ 2246.332490] Key type ._llcrypt unregistered [ 2253.034862] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2253.041259] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2255.770531] Key type ._llcrypt registered [ 2255.772345] Key type .llcrypt registered [ 2255.944811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2255.969513] alg: No test for adler32 (adler32-zlib) [ 2256.916662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2256.998773] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2269.786862] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 21:23:00 (1788484980) [ 2270.226281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2270.271360] LNet: 104226:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2270.279502] LNetError: 104226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2270.307482] LNet: Removed LNI 10.1.2.3@tcp [ 2270.335701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2271.347975] Key type .llcrypt unregistered [ 2271.350844] Key type ._llcrypt unregistered [ 2280.055587] Key type ._llcrypt registered [ 2280.057273] Key type .llcrypt registered [ 2280.300327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2280.322341] alg: No test for adler32 (adler32-zlib) [ 2281.356554] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 2281.433487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.5@tcp [ 2281.593574] LNet: 104510:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2281.613623] LNetError: 104510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2282.662875] LNet: Removed LNI 192.168.201.5@tcp [ 2283.649584] Key type .llcrypt unregistered [ 2283.658669] Key type ._llcrypt unregistered [ 2300.903263] Key type ._llcrypt registered [ 2300.909878] Key type .llcrypt registered [ 2302.627701] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 21:23:32 (1788485012) [ 2303.566401] Key type .llcrypt unregistered [ 2303.567601] Key type ._llcrypt unregistered [ 2312.029773] Key type ._llcrypt registered [ 2312.049146] Key type .llcrypt registered [ 2312.505599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2312.530798] alg: No test for adler32 (adler32-zlib) [ 2319.611545] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 21:23:50 (1788485030) [ 2319.782669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2319.830661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2319.871859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2319.905791] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 2320.649715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2320.675219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2320.701213] LNet: Removed LNI 192.168.201.5@tcp [ 2321.007343] LNet: 106250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2321.411604] Key type .llcrypt unregistered [ 2321.413235] Key type ._llcrypt unregistered [ 2332.638181] Key type ._llcrypt registered [ 2332.639893] Key type .llcrypt registered [ 2334.065607] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 21:24:04 (1788485044) [ 2334.596776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2334.608853] alg: No test for adler32 (adler32-zlib) [ 2335.757572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2335.863801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2335.912923] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 2336.820712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.5@tcp [ 2336.856458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.5@tcp [ 2336.897313] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2336.904255] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2336.913233] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2336, 0, 0 [ 2336.922883] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2336.927298] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2336.934204] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2336.938927] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2336, 0, 0 [ 2336.946574] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2336.951971] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2336.961815] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2336.967228] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2336, 0, 2386 [ 2336.975628] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2336.979954] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2336.986236] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2336.989915] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2336, 0, 2386 [ 2336.997421] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2344.112785] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 21:24:14 (1788485054) [ 2344.805303] LNet: 107787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2344.816704] LNetError: 107787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2344.837673] LNet: Removed LNI 192.168.201.5@tcp [ 2345.447597] Key type .llcrypt unregistered [ 2345.451525] Key type ._llcrypt unregistered [ 2353.967972] Key type ._llcrypt registered [ 2353.973733] Key type .llcrypt registered [ 2354.307521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2354.319509] alg: No test for adler32 (adler32-zlib) [ 2355.237108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2355.260408] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 2364.711585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.105@tcp [ 2374.186885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2374.284775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2374.305656] LNet: Added LNI 192.168.201.5@tcp1 [8/256/0/180] [ 2375.989711] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2376.198607] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2376.211911] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2376.243400] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2378.998180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2379.061952] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2393.462604] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 21:25:03 (1788485103) [ 2393.645756] LNet: 109286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2393.663646] LNetError: 109286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2393.670794] LNet: Removed LNI 192.168.201.5@tcp [ 2394.720515] LNet: Removed LNI 192.168.201.5@tcp1 [ 2394.732042] LNetError: 109286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2394.745118] LNetError: 109286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2394.762967] LNet: Removed LNI 10.1.2.3@tcp99 [ 2395.623187] Key type .llcrypt unregistered [ 2395.625884] Key type ._llcrypt unregistered [ 2403.185141] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2403.203896] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2404.795187] Key type ._llcrypt registered [ 2404.801543] Key type .llcrypt registered [ 2405.217303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2405.290484] alg: No test for adler32 (adler32-zlib) [ 2406.404816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2406.501755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2406.548800] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 2408.005416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2408.034378] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2411.356358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2411.408377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2411.497082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2411.545391] LNetError: 110029:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2411.593313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2411.649362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.5@tcp [ 2411.951321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2411.997979] LNet: 110049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2412.002515] LNetError: 110049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2412.007797] LNet: Removed LNI 10.1.2.3@tcp [ 2413.031077] LNet: Removed LNI 192.168.201.5@tcp [ 2413.653502] Key type .llcrypt unregistered [ 2413.655497] Key type ._llcrypt unregistered [ 2427.968702] Key type ._llcrypt registered [ 2427.972674] Key type .llcrypt registered [ 2429.381433] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 21:25:39 (1788485139) [ 2430.102904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2430.110817] alg: No test for adler32 (adler32-zlib) [ 2430.970880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2431.008691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2431.024959] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 2431.599720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.5@tcp [ 2431.672174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg105-client@tcp [ 2439.356839] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 21:25:49 (1788485149) [ 2439.653966] LNet: 111581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2439.661906] LNetError: 111581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2440.679484] LNet: Removed LNI 192.168.201.5@tcp [ 2441.673551] Key type .llcrypt unregistered [ 2441.676361] Key type ._llcrypt unregistered [ 2448.363338] Key type ._llcrypt registered [ 2448.366928] Key type .llcrypt registered [ 2448.586727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2448.635062] alg: No test for adler32 (adler32-zlib) [ 2449.637786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2449.688704] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 2449.730295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2457.430211] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 21:26:07 (1788485167) [ 2459.774216] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2461.617706] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 21:26:12 (1788485172) [ 2463.721723] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2466.232481] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 21:26:16 (1788485176) [ 2466.322616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2466.367946] LNet: 112996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2466.375738] LNetError: 112996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2466.398094] LNet: Removed LNI 192.168.201.5@tcp [ 2466.421624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2466.499142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2466.545444] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 2467.595624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2467.705950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2475.407563] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 21:26:25 (1788485185) [ 2475.560134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2475.583303] LNet: 113705:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2475.596745] LNetError: 113705:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2475.620344] LNet: Removed LNI 192.168.201.5@tcp [ 2475.649561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2475.752659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2475.769485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2476.131803] Key type .llcrypt unregistered [ 2476.133122] Key type ._llcrypt unregistered [ 2489.692380] Key type ._llcrypt registered [ 2489.703400] Key type .llcrypt registered [ 2491.525731] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 21:26:41 (1788485201) [ 2491.923209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2491.945630] alg: No test for adler32 (adler32-zlib) [ 2492.920180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2492.994149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2493.040594] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 2493.853917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2493.904909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.5@tcp [ 2501.682914] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 21:26:52 (1788485212) [ 2501.996343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2502.048088] LNet: 115264:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2502.060987] LNetError: 115264:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2502.081778] LNet: Removed LNI 192.168.201.5@tcp [ 2502.136633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2502.249448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2502.285211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2509.531389] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 21:26:59 (1788485219) [ 2509.618032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2509.646924] LNet: 115837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2509.675952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2509.794365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2510.248244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2510.278928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2510.359686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2510.888061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2510.925265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2511.004180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2511.023786] LNet: 115877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2511.047797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2511.169436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2511.800322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2511.870236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2511.941767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2512.648895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2512.726187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2519.719666] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 21:27:10 (1788485230) [ 2519.957115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2519.989994] LNet: 116477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2520.096523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2520.175954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2520.194088] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 2521.550240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2521.573189] LNet: Added LNI 192.168.201.5@tcp2 [8/256/0/180] [ 2522.596536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2522.681054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.5@tcp [ 2523.011483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2523.045826] LNetError: 116758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2524.064279] LNet: Removed LNI 192.168.201.5@tcp [ 2524.073891] LNetError: 116758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2524.108497] LNet: Removed LNI 192.168.201.5@tcp2 [ 2524.170836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2524.269875] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 2524.276109] LNet: Added LNI 192.168.201.5@tcp2 [8/256/0/180] [ 2524.286180] LNet: 116764:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2524.931927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.5@tcp [ 2525.250984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2525.275650] LNet: 116802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2525.281411] LNet: 116802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2526.306725] LNet: Removed LNI 192.168.201.5@tcp [ 2526.308342] LNetError: 116802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2526.319310] LNetError: 116802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2526.338600] LNet: Removed LNI 192.168.201.5@tcp2 [ 2526.378091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2526.461508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2526.504962] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 2527.221811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2527.238147] LNet: Added LNI 192.168.201.5@tcp2 [8/256/0/180] [ 2528.265614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.5@tcp [ 2528.332738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2528.417881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.5@tcp [ 2528.814873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2529.888357] LNet: Removed LNI 192.168.201.5@tcp [ 2530.921363] LNet: Removed LNI 192.168.201.5@tcp2 [ 2530.945277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2531.047284] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 2531.051844] LNet: Added LNI 192.168.201.5@tcp2 [8/256/0/180] [ 2531.056767] LNet: 117092:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2531.657812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.5@tcp [ 2531.863126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2531.873749] LNetError: 117130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2531.876976] LNetError: 117130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2531.880425] LNet: Removed LNI 192.168.201.5@tcp [ 2532.903838] LNet: Removed LNI 192.168.201.5@tcp2 [ 2532.985637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2533.059551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2533.107636] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 2534.095174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2534.161946] LNet: Added LNI 192.168.201.5@tcp2 [8/256/0/180] [ 2535.209275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2535.273616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.5@tcp [ 2535.564683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2535.599818] LNet: 117409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2535.612205] LNet: 117409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2536.672820] LNet: Removed LNI 192.168.201.5@tcp [ 2536.694258] LNet: Removed LNI 192.168.201.5@tcp2 [ 2536.762233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2536.874288] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 2536.882442] LNet: Added LNI 192.168.201.5@tcp2 [8/256/0/180] [ 2536.888630] LNet: 117415:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2537.515166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.5@tcp [ 2537.796749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2538.912306] LNet: Removed LNI 192.168.201.5@tcp [ 2538.924701] LNet: Removed LNI 192.168.201.5@tcp2 [ 2538.946767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2539.040389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2539.096714] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 2539.964510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2539.984256] LNet: Added LNI 192.168.201.5@tcp2 [8/256/0/180] [ 2541.033534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.5@tcp [ 2541.075926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2541.142522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.5@tcp [ 2541.399828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2541.445328] LNetError: 117737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2541.459350] LNetError: 117737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2541.477181] LNet: Removed LNI 192.168.201.5@tcp [ 2542.503685] LNet: Removed LNI 192.168.201.5@tcp2 [ 2542.549438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2542.661785] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 2542.672825] LNet: Added LNI 192.168.201.5@tcp2 [8/256/0/180] [ 2542.683404] LNet: 117743:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2543.532599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.5@tcp [ 2543.755267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2543.776502] LNet: Removed LNI 192.168.201.5@tcp [ 2544.808161] LNet: Removed LNI 192.168.201.5@tcp2 [ 2544.856698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2544.929481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2544.965802] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 2545.791627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2545.810610] LNet: Added LNI 192.168.201.5@tcp2 [8/256/0/180] [ 2546.736073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2546.779682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2546.863956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.5@tcp [ 2547.257518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2548.320401] LNet: Removed LNI 192.168.201.5@tcp [ 2548.354334] LNet: Removed LNI 192.168.201.5@tcp2 [ 2548.387918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2548.492644] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 2548.510158] LNet: Added LNI 192.168.201.5@tcp2 [8/256/0/180] [ 2548.518627] LNet: 118083:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2549.346554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.5@tcp [ 2549.873657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2550.944912] LNet: Removed LNI 192.168.201.5@tcp [ 2550.957954] LNet: Removed LNI 192.168.201.5@tcp2 [ 2551.036354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2551.083350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2551.100889] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 2551.826373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2551.864356] LNet: Added LNI 192.168.201.5@tcp2 [8/256/0/180] [ 2553.125422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.5@tcp [ 2553.201848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2553.280548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2553.417688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.5@tcp [ 2553.932633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2553.962483] LNet: 118438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2553.970101] LNet: 118438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2555.041131] LNet: Removed LNI 192.168.201.5@tcp [ 2556.071377] LNet: Removed LNI 192.168.201.5@tcp2 [ 2556.110364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2556.238104] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 2556.258251] LNet: Added LNI 192.168.201.5@tcp2 [8/256/0/180] [ 2557.069445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.5@tcp [ 2565.047638] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2566.625700] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 21:27:57 (1788485277) [ 2566.938544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2566.961401] LNetError: 119203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2566.970118] LNetError: 119203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 2566.976161] LNet: Removed LNI 192.168.201.5@tcp [ 2568.053312] LNet: Removed LNI 192.168.201.5@tcp2 [ 2568.128942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2568.691329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2568.729425] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 2569.660455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2569.675332] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2570.684677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2570.755374] LNet: Removed LNI 10.1.2.3@tcp [ 2570.772922] LNet: Removed LNI 192.168.201.5@tcp [ 2570.812939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2570.956788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2571.018186] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 2571.033425] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2571.718982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2571.746337] LNet: Removed LNI 192.168.201.5@tcp [ 2571.770614] LNet: Removed LNI 10.1.2.3@tcp [ 2578.645761] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 21:28:09 (1788485289) [ 2578.917283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2578.978405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2579.066170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2579.098463] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 2579.937587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2580.007252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.5@tcp [ 2580.054265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.5@tcp [ 2580.127446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.5@tcp [ 2580.210216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.5@tcp [ 2580.252443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.5@tcp [ 2580.303983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.5@tcp [ 2580.355736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.5@tcp [ 2580.394806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.5@tcp [ 2580.455472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.5@tcp [ 2580.490601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.5@tcp [ 2580.662321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2587.556559] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 21:28:18 (1788485298) [ 2587.773457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2587.790766] LNet: 120816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2587.797695] LNet: 120816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2588.839737] LNet: Removed LNI 192.168.201.5@tcp [ 2588.910846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2589.101872] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2589.199119] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2589.202230] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2589.209130] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2591.705838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2591.759520] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 2592.569961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2592.606222] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2594.932415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.5@tcp [ 2594.990460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2595.072627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.5@tcp -m PUT -e local_timeout -r 1 [ 2595.130735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2595.220755] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2595.236239] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2595.493619] LNetError: 120883:0:(lib-move.c:4801:lnet_parse()) 192.168.201.5@tcp, src 192.168.201.5@tcp: Dropping PUT (error -108 looking up sender) [ 2596.522917] LNet: Removed LNI 10.1.2.3@tcp [ 2597.549427] LNet: Removed LNI 192.168.201.5@tcp [ 2597.988486] Key type .llcrypt unregistered [ 2597.991453] Key type ._llcrypt unregistered [ 2612.213689] Key type ._llcrypt registered [ 2612.216952] Key type .llcrypt registered [ 2613.781888] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 21:28:44 (1788485324) [ 2614.399948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2614.438064] alg: No test for adler32 (adler32-zlib) [ 2615.340507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2615.815142] LNet: 122192:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2616.557487] Key type .llcrypt unregistered [ 2616.561024] Key type ._llcrypt unregistered [ 2623.639467] Key type ._llcrypt registered [ 2623.642182] Key type .llcrypt registered [ 2624.061462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2624.085588] alg: No test for adler32 (adler32-zlib) [ 2625.045376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2625.087109] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 2635.399674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.105@tcp [ 2635.644923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2636.293892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.105@tcp [ 2636.385123] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.105@tcp [ 2636.556521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2636.599752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2636.630593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2636.653894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2636.678815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2636.715768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2636.755670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2636.785352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2636.807529] LNet: 122640:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2636.836567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2636.864336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2636.905950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2636.939784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2636.969533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2637.001691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2637.722246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2638.009082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2638.076671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2638.140694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2638.223938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2638.256933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2638.301536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2638.328793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2638.356192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2638.373632] LNet: 122757:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2638.394818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2638.421601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2638.448765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2638.480317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2638.512698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2638.531237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2639.314394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.105@tcp [ 2639.617406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2639.665877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2639.718781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2639.770183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2639.808493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2639.862738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2639.908490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2639.959609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2639.986220] LNet: 122875:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2640.013375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2640.066716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2640.124594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2640.162747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2640.230031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2640.285636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2640.977959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2641.226799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2641.267346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2641.292328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2641.317886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2641.374553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2641.406700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2641.442994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2641.479916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2641.522136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2641.564981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2641.609248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2641.645043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2641.682880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2641.713170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2642.038566] LNet: 123038:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2642.044349] LNetError: 123038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2642.062065] LNet: Removed LNI 192.168.201.5@tcp [ 2642.538927] Key type .llcrypt unregistered [ 2642.544807] Key type ._llcrypt unregistered [ 2661.546558] Key type ._llcrypt registered [ 2661.548918] Key type .llcrypt registered [ 2663.063474] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 21:29:33 (1788485373) [ 2663.678732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2663.705213] alg: No test for adler32 (adler32-zlib) [ 2664.644771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2665.114795] LNet: 123924:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2665.820227] Key type .llcrypt unregistered [ 2665.821966] Key type ._llcrypt unregistered [ 2674.681331] Key type ._llcrypt registered [ 2674.682809] Key type .llcrypt registered [ 2674.842674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2674.849331] alg: No test for adler32 (adler32-zlib) [ 2675.862478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2675.937748] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 2685.337466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.105@tcp [ 2696.128960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2696.213515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2696.276693] LNet: Added LNI 192.168.201.5@tcp1 [8/256/0/180] [ 2697.549716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2698.338709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.105@tcp [ 2698.464089] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.105@tcp [ 2698.675186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2698.735491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2698.786808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2698.840878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2698.902772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2698.958446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2699.017596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2699.064255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2699.092430] LNet: 124575:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2699.117908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2699.156188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2699.182817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2699.212923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2699.243425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2699.270185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2699.899094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2700.179789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2700.209366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2700.244869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2700.279966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2700.318283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2700.354462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2700.390139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2700.431431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2700.461922] LNet: 124692:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2700.498094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2700.557905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2700.594138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2700.629814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2700.688161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2700.734929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2701.475213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.105@tcp [ 2701.669355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2701.698352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2701.714670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2701.743054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2701.778890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2701.819851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2701.870488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2701.936184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2701.965469] LNet: 124810:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2701.980740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2702.022782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2702.074196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2702.108289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2702.136711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2702.164771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2702.675682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2702.884728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2702.908277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2702.932869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2702.971617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2702.995627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2703.028912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2703.054350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2703.087705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2703.111434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2703.137877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2703.174611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2703.221886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2703.270986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2703.334442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2708.446635] LNet: 125008:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2708.453894] LNetError: 125008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2708.461992] LNet: Removed LNI 192.168.201.5@tcp [ 2708.490501] LNet: Removed LNI 192.168.201.5@tcp1 [ 2709.080866] Key type .llcrypt unregistered [ 2709.085136] Key type ._llcrypt unregistered [ 2726.867963] Key type ._llcrypt registered [ 2726.870818] Key type .llcrypt registered [ 2727.872232] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 21:30:38 (1788485438) [ 2728.386416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2728.401251] alg: No test for adler32 (adler32-zlib) [ 2729.357450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2729.451671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2729.485432] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 2730.273616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2730.303162] LNet: Added LNI 192.168.201.5@tcp2 [8/256/0/180] [ 2730.899808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.5@tcp [ 2731.017391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2732.064137] LNet: Removed LNI 192.168.201.5@tcp [ 2741.150029] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 21:30:50 (1788485450) [ 2742.980275] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2745.441233] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 21:30:55 (1788485455) [ 2754.656592] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 21:31:04 (1788485464) [ 2767.348905] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 21:31:17 (1788485477) [ 2773.851416] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 21:31:24 (1788485484) [ 2782.418932] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 21:31:32 (1788485492) [ 2791.296453] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 21:31:41 (1788485501) [ 2800.387911] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 21:31:50 (1788485510) [ 2809.305340] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 21:31:59 (1788485519) [ 2817.744338] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 21:32:08 (1788485528) [ 2825.390025] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 21:32:16 (1788485536) [ 2832.531269] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 21:32:23 (1788485543) [ 2840.550930] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 21:32:30 (1788485550) [ 2847.959524] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 21:32:38 (1788485558) [ 2856.068503] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 21:32:46 (1788485566) [ 2864.343646] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 21:32:54 (1788485574) [ 2873.289097] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 21:33:03 (1788485583) [ 2882.206627] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 21:33:12 (1788485592) [ 2890.813362] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 21:33:21 (1788485601) [ 2891.017225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2891.057388] LNet: 136706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2891.077177] LNetError: 136706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2892.144173] LNet: Removed LNI 192.168.201.5@tcp2 [ 2892.199630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2892.363031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2892.384329] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 2893.637317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2893.654722] LNet: 136851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2893.659989] LNetError: 136851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2893.675450] LNet: Removed LNI 192.168.201.5@tcp [ 2893.693283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2893.772773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2893.834738] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 2902.009665] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 21:33:31 (1788485611) [ 2913.483887] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 21:33:43 (1788485623) [ 2921.885660] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 21:33:52 (1788485632) [ 2930.145151] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 21:34:00 (1788485640) [ 2937.560270] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 21:34:08 (1788485648) [ 2944.637980] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 21:34:15 (1788485655) [ 2952.442641] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 21:34:23 (1788485663) [ 2960.317663] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 21:34:30 (1788485670) [ 2968.716928] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 21:34:39 (1788485679) [ 2977.076617] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 21:34:47 (1788485687) [ 2985.708434] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 21:34:55 (1788485695) [ 2994.656862] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 21:35:05 (1788485705) [ 3002.828546] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2736 sec ========== 21:35:13 (1788485713) [ 3003.310575] LNet: 144317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3003.315955] LNetError: 144317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3003.333319] LNet: Removed LNI 192.168.201.5@tcp [ 3004.031951] Key type .llcrypt unregistered [ 3004.035977] Key type ._llcrypt unregistered [ 3019.175706] Key type ._llcrypt registered [ 3019.197874] Key type .llcrypt registered [ 3019.750737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3019.763245] alg: No test for adler32 (adler32-zlib) [ 3020.874386] Lustre: Lustre: Build Version: 2.17.57_84_g5b8c505 [ 3021.285698] LNet: Added LNI 192.168.201.5@tcp [8/256/0/180] [ 3023.008276] Key type lgssc registered [ 3025.195437] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3098.678955] Lustre: Mounted lustre-client - version 2.17.57_84_g5b8c505 [ 3104.963455] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3114.440905] Lustre: Unmounted lustre-client [ 3141.448546] Key type lgssc unregistered [ 3141.936792] LNet: 147267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3141.947543] LNetError: 147267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3141.975587] LNet: Removed LNI 192.168.201.5@tcp [ 3143.460204] Key type .llcrypt unregistered [ 3143.462541] Key type ._llcrypt unregistered