[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 504343607 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003154] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008533] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011140] LSM: Security Framework initializing [ 0.012048] Yama: becoming mindful. [ 0.013035] SELinux: Initializing. [ 0.014087] *** VALIDATE selinux *** [ 0.022592] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027370] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028177] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029109] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031075] *** VALIDATE tmpfs *** [ 0.032475] *** VALIDATE proc *** [ 0.033276] *** VALIDATE cgroup *** [ 0.034012] *** VALIDATE cgroup2 *** [ 0.035306] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037147] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039031] Spectre V2 : User space: Vulnerable [ 0.040011] Speculative Store Bypass: Vulnerable [ 0.043251] debug: unmapping init [mem 0xffffffff8c659000-0xffffffff8c660fff] [ 0.046000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046687] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047026] ... version: 2 [ 0.048013] ... bit width: 48 [ 0.049011] ... generic registers: 4 [ 0.050012] ... value mask: 0000ffffffffffff [ 0.051015] ... max period: 00007fffffffffff [ 0.052012] ... fixed-purpose events: 3 [ 0.053010] ... event mask: 000000070000000f [ 0.054305] rcu: Hierarchical SRCU implementation. [ 0.056470] smp: Bringing up secondary CPUs ... [ 0.057586] x86: Booting SMP configuration: [ 0.058028] .... node #0, CPUs: #1 #2 #3 [ 0.070704] smp: Brought up 1 node, 4 CPUs [ 0.072013] smpboot: Max logical packages: 1 [ 0.073011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.109206] node 0 deferred pages initialised in 35ms [ 0.112339] devtmpfs: initialized [ 0.113271] x86/mm: Memory block size: 128MB [ 0.115974] gcov: version magic: 0x41383552 [ 0.117333] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.118132] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.119319] pinctrl core: initialized pinctrl subsystem [ 0.120209] [ 0.120852] ************************************************************* [ 0.121020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.122016] ** ** [ 0.123016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.124013] ** ** [ 0.125018] ** This means that this kernel is built to expose internal ** [ 0.126016] ** IOMMU data structures, which may compromise security on ** [ 0.127013] ** your system. ** [ 0.128016] ** ** [ 0.129013] ** If you see this message and you are not debugging the ** [ 0.130016] ** kernel, report this immediately to your vendor! ** [ 0.131014] ** ** [ 0.132016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.133015] ************************************************************* [ 0.134684] NET: Registered protocol family 16 [ 0.135451] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.136061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.137057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.138601] cpuidle: using governor menu [ 0.140548] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.143512] PCI: Using configuration type 1 for base access [ 0.145121] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.156068] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.158024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.162098] cryptd: max_cpu_qlen set to 1000 [ 0.164260] ACPI: Added _OSI(Module Device) [ 0.165015] ACPI: Added _OSI(Processor Device) [ 0.166011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.167011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.171800] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.173677] ACPI: Interpreter enabled [ 0.174062] ACPI: PM: (supports S0 S3 S4 S5) [ 0.175009] ACPI: Using IOAPIC for interrupt routing [ 0.176112] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.177383] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.186949] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.187040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.188018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.189092] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.191442] acpiphp: Slot [2] registered [ 0.192119] acpiphp: Slot [5] registered [ 0.193092] acpiphp: Slot [6] registered [ 0.194077] acpiphp: Slot [3] registered [ 0.195069] acpiphp: Slot [4] registered [ 0.196083] acpiphp: Slot [7] registered [ 0.197108] acpiphp: Slot [8] registered [ 0.198109] acpiphp: Slot [9] registered [ 0.199139] acpiphp: Slot [10] registered [ 0.200078] acpiphp: Slot [11] registered [ 0.201086] acpiphp: Slot [12] registered [ 0.202112] acpiphp: Slot [13] registered [ 0.203103] acpiphp: Slot [14] registered [ 0.204115] acpiphp: Slot [15] registered [ 0.205120] acpiphp: Slot [16] registered [ 0.206101] acpiphp: Slot [17] registered [ 0.207096] acpiphp: Slot [18] registered [ 0.208072] acpiphp: Slot [19] registered [ 0.209085] acpiphp: Slot [20] registered [ 0.210097] acpiphp: Slot [21] registered [ 0.211117] acpiphp: Slot [22] registered [ 0.212096] acpiphp: Slot [23] registered [ 0.213115] acpiphp: Slot [24] registered [ 0.214099] acpiphp: Slot [25] registered [ 0.215085] acpiphp: Slot [26] registered [ 0.216093] acpiphp: Slot [27] registered [ 0.217108] acpiphp: Slot [28] registered [ 0.218103] acpiphp: Slot [29] registered [ 0.219082] acpiphp: Slot [30] registered [ 0.220035] acpiphp: Slot [31] registered [ 0.221031] PCI host bridge to bus 0000:00 [ 0.222013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.223051] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.224023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.225029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.226077] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.229029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.231175] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.234044] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.237392] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.245013] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.249058] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.253021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.255014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.257015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.259565] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.262942] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.265045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.268771] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.271781] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.282019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.287015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.293729] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.304023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.314017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.336019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.346395] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.351015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.357020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.371014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.378735] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.381409] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.384347] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.386357] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.389199] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.396029] iommu: Default domain type: Passthrough [ 0.397387] SCSI subsystem initialized [ 0.398091] ACPI: bus type USB registered [ 0.399038] usbcore: registered new interface driver usbfs [ 0.400043] usbcore: registered new interface driver hub [ 0.401069] usbcore: registered new device driver usb [ 0.402090] pps_core: LinuxPPS API ver. 1 registered [ 0.403006] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.405052] PTP clock support registered [ 0.406101] EDAC MC: Ver: 3.0.0 [ 0.407475] PCI: Using ACPI for IRQ routing [ 0.408484] NetLabel: Initializing [ 0.409007] NetLabel: domain hash size = 128 [ 0.409852] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.410024] NetLabel: unlabeled traffic allowed by default [ 0.416178] vgaarb: loaded [ 0.418264] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.419014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.426130] clocksource: Switched to clocksource kvm-clock [ 0.530439] VFS: Disk quotas dquot_6.6.0 [ 0.531767] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.533419] *** VALIDATE ramfs *** [ 0.534175] *** VALIDATE hugetlbfs *** [ 0.535127] pnp: PnP ACPI init [ 0.536898] pnp: PnP ACPI: found 6 devices [ 0.557169] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.560341] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.562340] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.564342] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.566653] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.568583] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.571284] NET: Registered protocol family 2 [ 0.573351] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.578204] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.582759] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.589711] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.593857] TCP: Hash tables configured (established 65536 bind 65536) [ 0.597833] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.601188] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.604061] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.607688] NET: Registered protocol family 1 [ 0.609878] RPC: Registered named UNIX socket transport module. [ 0.612229] RPC: Registered udp transport module. [ 0.614482] RPC: Registered tcp transport module. [ 0.616508] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.619682] NET: Registered protocol family 44 [ 0.621671] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.624191] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.627518] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.630471] PCI: CLS 0 bytes, default 64 [ 0.632653] Unpacking initramfs... [ 3.310744] debug: unmapping init [mem 0xffff9cbebcc64000-0xffff9cbebffcffff] [ 3.330212] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.332638] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.345119] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.299144] Initialise system trusted keyrings [ 5.315513] Key type blacklist registered [ 5.323730] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.346654] zbud: loaded [ 5.352510] *** VALIDATE nfs *** [ 5.355371] *** VALIDATE nfs4 *** [ 5.360522] pstore: using deflate compression [ 5.382126] Platform Keyring initialized [ 5.775574] NET: Registered protocol family 38 [ 5.782964] Key type asymmetric registered [ 5.793234] Asymmetric key parser 'x509' registered [ 5.803164] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.809943] io scheduler mq-deadline registered [ 5.812188] io scheduler kyber registered [ 5.813819] io scheduler bfq registered [ 5.819738] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.825270] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.829549] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.834298] ACPI: Power Button [PWRF] [ 5.843612] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.852174] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.867416] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.911297] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.987707] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.021308] Non-volatile memory driver v1.3 [ 6.028792] Linux agpgart interface v0.103 [ 6.109034] virtio_blk virtio1: [vda] 149408 512-byte logical blocks (76.5 MB/73.0 MiB) [ 6.126110] vda: detected capacity change from 0 to 76496896 [ 6.173790] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.181670] vdb: detected capacity change from 0 to 1073741824 [ 6.202620] libphy: Fixed MDIO Bus: probed [ 6.237717] usbcore: registered new interface driver usbserial_generic [ 6.242578] usbserial: USB Serial support registered for generic [ 6.249988] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.257096] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.259039] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.286640] mousedev: PS/2 mouse device common for all mice [ 6.300996] rtc_cmos 00:05: RTC can wake from S4 [ 6.325432] rtc_cmos 00:05: registered as rtc0 [ 6.331120] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.334049] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.346605] intel_pstate: CPU model not supported [ 6.357299] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.362103] hid: raw HID events driver (C) Jiri Kosina [ 6.371862] usbcore: registered new interface driver usbhid [ 6.374708] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.376164] usbhid: USB HID core driver [ 6.384327] drop_monitor: Initializing network drop monitor service [ 6.390209] Initializing XFRM netlink socket [ 6.398041] NET: Registered protocol family 10 [ 6.402236] Segment Routing with IPv6 [ 6.403907] NET: Registered protocol family 17 [ 6.409694] mpls_gso: MPLS GSO support [ 6.427905] RAS: Correctable Errors collector initialized. [ 6.433262] AVX version of gcm_enc/dec engaged. [ 6.436703] AES CTR mode by8 optimization enabled [ 6.657927] sched_clock: Marking stable (6657895337, 0)->(7629201305, -971305968) [ 6.683229] registered taskstats version 1 [ 6.691865] Loading compiled-in X.509 certificates [ 6.696165] zswap: loaded using pool lzo/zbud [ 6.768409] Key type big_key registered [ 6.803338] Key type encrypted registered [ 6.805095] ima: No TPM chip found, activating TPM-bypass! [ 6.816255] ima: Allocated hash algorithm: sha1 [ 6.822911] ima: No architecture policies found [ 6.827767] evm: Initialising EVM extended attributes: [ 6.830338] evm: security.selinux [ 6.833477] evm: security.ima [ 6.834903] evm: security.capability [ 6.838731] evm: HMAC attrs: 0x1 [ 6.852409] rtc_cmos 00:05: setting system clock to 2026-08-22 08:35:20 UTC (1787387720) [ 6.867532] debug: unmapping init [mem 0xffffffff8d603000-0xffffffff8d7fffff] [ 6.882912] debug: unmapping init [mem 0xffffffff8c382000-0xffffffff8c658fff] [ 6.907155] Write protecting the kernel read-only data: 28672k [ 6.911465] debug: unmapping init [mem 0xffffffff8aa03000-0xffffffff8abfffff] [ 6.916849] debug: unmapping init [mem 0xffffffff8b314000-0xffffffff8b3fffff] [ 7.013090] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.033844] systemd[1]: Detected virtualization kvm. [ 7.040623] systemd[1]: Detected architecture x86-64. [ 7.046161] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.124592] systemd[1]: No hostname configured. [ 7.127445] systemd[1]: Set hostname to . [ 7.131985] random: systemd: uninitialized urandom read (16 bytes read) [ 7.136251] systemd[1]: Initializing machine ID from random generator. [ 7.219938] random: ln: uninitialized urandom read (6 bytes read) [ 7.451299] random: systemd: uninitialized urandom read (16 bytes read) [ 7.453571] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 7.471972] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 7.494717] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 9.128713] device-mapper: uevent: version 1.0.3 [ 9.133948] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Mounted Kernel Configuration File System. [ 10.989478] scsi host0: ata_piix [ 10.990419] virtio_net virtio0 ens2: renamed from eth0 [ 11.101799] scsi host1: ata_piix [ 11.103895] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 11.107633] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 11.486486] random: fast init done [ 16.368284] random: crng init done [ 16.371157] random: 7 urandom warning(s) missed due to ratelimiting [ 16.545988] dracut-initqueue[571]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 18.832180] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 21.218250] printk: systemd: 26 output lines suppressed due to ratelimiting [ 22.233498] SELinux: Disabled at runtime. [ 22.426429] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 22.448641] systemd[1]: Detected virtualization kvm. [ 22.459138] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 24.715325] systemd[1]: initrd-switch-root.service: Succeeded. [ 24.729291] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 24.765411] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.776662] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.788915] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 24.815768] systemd[1]: Starting Journal Service... Starting Journal Service... [ 24.829353] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... [ 25.040539] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting udev Coldplug all Devices... [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 26.169448] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 27.453947] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 27.526960] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 28.037090] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 28.122048] EDAC sbridge: Ver: 1.1.2 [ 31.324498] Key type dns_resolver registered [ 32.005848] NFS: Registering the id_resolver key type [ 32.011829] Key type id_resolver registered [ 32.013826] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg260-client login: [ 62.267659] hrtimer: interrupt took 6649972 ns [ 97.619518] libcfs: loading out-of-tree module taints kernel. [ 97.798175] Key type ._llcrypt registered [ 97.802117] Key type .llcrypt registered [ 98.238449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 98.251896] alg: No test for adler32 (adler32-zlib) [ 99.848511] Lustre: Lustre: Build Version: 2.17.54_110_g2f98931 [ 100.713407] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 102.455777] Key type lgssc registered [ 104.236146] Lustre: Echo OBD driver; http://www.lustre.org/ [ 233.849582] Lustre: Mounted lustre-client [ 239.120797] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 256.448939] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing check_logdir /tmp/testlogs/ [ 259.551185] Lustre: lustre-OST0000-osc-ffff9cbf0a2a3000: disconnect after 23s idle [ 262.886810] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing yml_node [ 268.573786] Lustre: DEBUG MARKER: Client: 2.17.54.110 [ 272.090724] Lustre: DEBUG MARKER: MDS: 2.17.54.110 [ 275.314163] Lustre: DEBUG MARKER: OSS: 2.17.54.110 [ 276.855709] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sat Aug 22 04:39:49 EDT 2026 [ 296.051330] Lustre: Unmounted lustre-client [ 327.696123] Key type lgssc unregistered [ 328.139662] LNet: 8392:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 328.159449] LNetError: 8392:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 329.198269] LNet: Removed LNI 192.168.202.60@tcp [ 330.425162] Key type .llcrypt unregistered [ 330.434606] Key type ._llcrypt unregistered [ 363.818420] Key type ._llcrypt registered [ 363.828174] Key type .llcrypt registered [ 363.899546] LNet: 8972:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 363.899546] mask usage: [+|-] ... [ 364.147804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 364.174273] alg: No test for adler32 (adler32-zlib) [ 365.135500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 365.208645] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 365.241733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 369.982835] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 370.080175] LNet: 9270:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 370.085266] LNetError: 9270:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 370.116260] LNet: Removed LNI 192.168.202.60@tcp [ 370.752377] Key type .llcrypt unregistered [ 370.759202] Key type ._llcrypt unregistered [ 378.924863] Key type ._llcrypt registered [ 378.926796] Key type .llcrypt registered [ 380.867886] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 04:41:32 (1787388092) [ 381.232545] LNet: 9646:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 381.232545] mask usage: [+|-] ... [ 381.538527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 381.567966] alg: No test for adler32 (adler32-zlib) [ 382.582737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 382.726748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 382.752694] LNet: 9690:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 382.785575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 382.885898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 382.913905] LNet: 9697:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 394.011109] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 04:41:45 (1787388105) [ 394.157813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 394.206804] LNet: 10276:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 394.237434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.380221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 403.535991] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 04:41:55 (1787388115) [ 403.621548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 403.632242] LNet: 10859:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 403.651864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 403.715720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 410.895670] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 04:42:02 (1787388122) [ 411.043022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 411.084253] LNet: 11443:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 411.100399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 411.196034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 419.969745] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 04:42:11 (1787388131) [ 420.041645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 420.069622] LNet: 12029:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 420.093329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 420.217332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 420.838290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 428.854595] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 04:42:20 (1787388140) [ 428.973601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.019322] LNet: 12618:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 429.057281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.248891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 429.996486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 430.619356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 438.092787] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 04:42:30 (1787388150) [ 438.376496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 438.456938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 438.577823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 445.245763] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 04:42:37 (1787388157) [ 445.422964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 445.454100] LNet: 13804:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 445.461049] LNet: 13804:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 445.498889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 445.588047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 445.662490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 446.302785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 446.346312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 446.904586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 446.954908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 447.298670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 447.326426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 447.694886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 447.744399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 448.415529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 448.472717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 449.086471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 449.127602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 449.686406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 449.721433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 450.244477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 450.299587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 450.760665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 450.811648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 451.254951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 451.286439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 451.877677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 451.946195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 452.351809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 452.396527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 459.523930] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 04:42:51 (1787388171) [ 459.651068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 459.684160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 459.769610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 459.801950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 466.365114] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 04:42:58 (1787388178) [ 466.776894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 466.825551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 466.909006] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 466.946176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 474.084657] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 04:43:06 (1787388186) [ 474.249019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 474.301624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 474.389826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 474.443066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 481.415938] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 04:43:13 (1787388193) [ 481.652252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 481.703339] LNet: 16247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 481.719295] LNet: 16247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 481.751961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 481.880370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 481.910843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 488.405780] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 04:43:20 (1787388200) [ 488.514230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 488.575225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 488.709965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 488.739985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 495.787683] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 04:43:28 (1787388208) [ 495.932079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 496.016902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.141219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 496.231061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 505.355571] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 04:43:37 (1787388217) [ 505.495222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.571226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.711668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 506.374580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 515.180475] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 04:43:46 (1787388226) [ 515.505748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 515.557275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 515.871706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 516.257085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 523.534380] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 04:43:55 (1787388235) [ 523.623290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 523.692405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 523.878046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 524.133239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 531.812273] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 04:44:04 (1787388244) [ 531.930594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 532.007559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 532.573738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 532.896870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 542.414276] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 04:44:14 (1787388254) [ 542.833363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.936224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 543.076029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 543.174706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 551.524955] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 04:44:23 (1787388263) [ 551.826318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 551.869568] LNet: 21081:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 551.878186] LNet: 21081:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 551.923599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 552.089235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 552.193417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 561.313078] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 04:44:33 (1787388273) [ 561.379862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.453697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.537253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 568.526285] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 04:44:40 (1787388280) [ 568.627804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 568.741906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.919666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 577.482228] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 04:44:49 (1787388289) [ 577.607214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 577.666832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 577.782237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 578.542485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 587.035725] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 04:44:59 (1787388299) [ 587.494937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 587.568053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 587.686719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 587.726698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 596.103653] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 04:45:08 (1787388308) [ 596.226439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 596.309381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 596.437993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 596.482649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 603.575376] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 04:45:15 (1787388315) [ 603.901176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 603.980965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.058910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 604.119528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 613.699866] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 04:45:25 (1787388325) [ 614.032628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 614.090738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 614.172928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 614.211232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 623.003959] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 04:45:34 (1787388334) [ 623.295801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 623.333377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.418095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 623.487909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 623.508309] LNetError: 25809:0:(peer.c:2234:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 623.554190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 623.588711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 630.669659] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 04:45:42 (1787388342) [ 631.024818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.054989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.120932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.4z7n6d55VW [ 631.216379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.4z7n6d55VW [ 631.297340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.4z7n6d55VW [ 637.821111] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 04:45:49 (1787388349) [ 637.999321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.093257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.171284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 638.199935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 638.254114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 644.370645] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 04:45:56 (1787388356) [ 644.520272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 644.579167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 644.693633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 648.463063] LNet: Message forwarding will be enabled in 2s [ 651.551145] LNet: Message forwarding enabled [ 651.997299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 652.045921] LNet: Message forwarding will stop in 2s to 6s [ 656.767745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 657.745926] LNet: Message forwarding disabled [ 659.638803] LNet: Message forwarding will be enabled in 2s [ 662.887326] LNet: Message forwarding enabled [ 668.147990] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 04:46:20 (1787388380) [ 668.215595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 669.959260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.038373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 676.959151] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 04:46:29 (1787388389) [ 677.154301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 677.244868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.294869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 677.316750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 677.351526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 677.382495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 677.414687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 677.455220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 677.497408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 677.541855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 677.595853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 677.652585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 677.692330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 677.736470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 677.777483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 677.805346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 677.868662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 677.907824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 677.945711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 678.012813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 678.062496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 678.107781] LNetError: 28881:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 684.189285] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 04:46:36 (1787388396) [ 684.495807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 684.539529] LNet: 29452:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 684.547479] LNet: 29452:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 13 previous similar messages [ 684.568961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 684.651582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 691.521734] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 04:46:43 (1787388403) [ 693.796210] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg260-server [ 695.365604] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 04:46:47 (1787388407) [ 695.491163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 695.579186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 695.663645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 695.708704] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 698.226758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[61-65]@tcp [ 698.268562] LNet: 30599:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 701.350828] LNetError: 30394:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.63@tcp has gone from up to down [ 704.482604] LNetError: 30392:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.65@tcp has gone from up to down [ 704.503512] LNetError: 30392:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 707.326528] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 04:46:59 (1787388419) [ 710.017697] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg260-server [ 710.576032] LNetError: 30384:0:(router.c:1266:lnet_check_routers()) Failed to discover router 192.168.202.62@tcp [ 711.989729] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 04:47:04 (1787388424) [ 712.377478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.678642] LNetError: 30384:0:(router.c:1266:lnet_check_routers()) Failed to discover router 192.168.202.65@tcp [ 712.712699] LNetError: 30384:0:(router.c:1266:lnet_check_routers()) Skipped 1 previous similar message [ 715.497272] LNetError: 31536:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 715.564308] LNet: Removed LNI 192.168.202.60@tcp [ 715.606452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 715.686896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 715.739481] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 717.038776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[61-66/2]@tcp [ 717.056706] LNet: 31654:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 717.063390] LNet: 31654:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 717.081871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[61-66/2]@tcp [ 724.064618] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 04:47:16 (1787388436) [ 724.200615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 724.264604] LNetError: 32240:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 724.285753] LNet: Removed LNI 192.168.202.60@tcp [ 724.310351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 724.430776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 724.477552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 724.520764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 724.576593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 724.641221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 724.705807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 724.773400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 724.842536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 724.899573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 724.994247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 725.156390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 725.207567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 725.256391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 725.390087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 725.455552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 725.595272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 725.714573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 725.769692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 725.820210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 725.901933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 725.951410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 726.018739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 726.088866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 726.132673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 726.179786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 726.288531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 726.337955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 726.430463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 726.533284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 726.607436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 726.685289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 726.789439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 726.847702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 726.935765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 735.020299] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 04:47:26 (1787388446) [ 737.391268] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg260-server [ 739.392078] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 04:47:31 (1787388451) [ 741.681172] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg260-server [ 744.194622] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 04:47:35 (1787388455) [ 745.194585] Key type .llcrypt unregistered [ 745.197124] Key type ._llcrypt unregistered [ 752.372609] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 752.633891] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 752.644279] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 752.656694] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 755.397789] Key type ._llcrypt registered [ 755.400274] Key type .llcrypt registered [ 755.439420] LNet: 33924:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 755.439420] mask usage: [+|-] ... [ 755.600661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 755.620569] alg: No test for adler32 (adler32-zlib) [ 756.616964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 756.683548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 756.729951] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 757.681924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 757.717670] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 759.869463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 759.890565] LNet: Removed LNI 10.1.2.3@tcp [ 760.018178] LNet: 34366:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 760.024194] LNetError: 34366:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 760.040672] LNet: Removed LNI 192.168.202.60@tcp [ 760.671581] Key type .llcrypt unregistered [ 760.673819] Key type ._llcrypt unregistered [ 773.369785] Key type ._llcrypt registered [ 773.372516] Key type .llcrypt registered [ 775.064536] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 04:48:06 (1787388486) [ 775.589986] Key type .llcrypt unregistered [ 775.593520] Key type ._llcrypt unregistered [ 782.546946] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 782.761120] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 782.772329] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 782.785545] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 785.425962] Key type ._llcrypt registered [ 785.428591] Key type .llcrypt registered [ 785.462237] LNet: 35405:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 785.462237] mask usage: [+|-] ... [ 785.627883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 785.639257] alg: No test for adler32 (adler32-zlib) [ 786.581632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 786.669198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 786.713722] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 787.828401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 787.862746] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 790.949949] LNet: 35863:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 790.955875] LNetError: 35863:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 790.959478] LNet: Removed LNI 10.1.2.3@tcp [ 790.993699] LNet: Removed LNI 192.168.202.60@tcp [ 791.811582] Key type .llcrypt unregistered [ 791.814683] Key type ._llcrypt unregistered [ 805.523756] Key type ._llcrypt registered [ 805.526780] Key type .llcrypt registered [ 807.144349] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 04:48:39 (1787388519) [ 807.662356] Key type .llcrypt unregistered [ 807.663876] Key type ._llcrypt unregistered [ 815.529979] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 815.812310] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 815.828731] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 815.847458] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 817.394624] Key type ._llcrypt registered [ 817.397682] Key type .llcrypt registered [ 817.452708] LNet: 36887:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 817.452708] mask usage: [+|-] ... [ 817.714466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 817.745368] alg: No test for adler32 (adler32-zlib) [ 818.706690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 818.764135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 818.796049] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 821.109208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 821.164891] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 823.779920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 823.805781] LNet: Removed LNI 10.1.2.3@tcp [ 823.833933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 823.912303] LNet: Removed LNI 10.1.2.31@tcp [ 823.944601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 823.992078] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 824.800745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 824.843540] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 826.279833] LNet: 37770:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 826.285566] LNetError: 37770:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 826.295955] LNet: Removed LNI 10.1.2.31@tcp [ 826.310494] LNet: Removed LNI 10.1.2.3@tcp [ 827.198397] Key type .llcrypt unregistered [ 827.204485] Key type ._llcrypt unregistered [ 839.443739] Key type ._llcrypt registered [ 839.449780] Key type .llcrypt registered [ 841.151367] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 04:49:12 (1787388552) [ 841.455675] LNet: 38563:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 841.455675] mask usage: [+|-] ... [ 841.637259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 841.660974] alg: No test for adler32 (adler32-zlib) [ 842.673953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 842.786708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 842.825798] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 843.797768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 843.839841] LNet: 38724:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 843.849764] LNetError: 38724:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 843.864990] LNet: Removed LNI 192.168.202.60@tcp [ 843.901686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 843.959647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.60@tcp [ 844.003714] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 845.231930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.60@tcp [ 845.290117] LNet: Removed LNI 192.168.202.60@tcp [ 845.386446] LNet: 38859:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 846.160086] Key type .llcrypt unregistered [ 846.162439] Key type ._llcrypt unregistered [ 859.290299] Key type ._llcrypt registered [ 859.291787] Key type .llcrypt registered [ 861.032651] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 04:49:32 (1787388572) [ 863.136997] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg260-server [ 864.874962] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 04:49:37 (1787388577) [ 865.388303] Key type .llcrypt unregistered [ 865.389792] Key type ._llcrypt unregistered [ 872.363122] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 872.585687] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 872.593945] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 872.619083] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 875.289576] Key type ._llcrypt registered [ 875.291817] Key type .llcrypt registered [ 875.314837] LNet: 40237:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 875.314837] mask usage: [+|-] ... [ 875.490487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 875.501691] alg: No test for adler32 (adler32-zlib) [ 876.456837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 876.529105] LNet: Added LNI 192.168.202.60@tcp1 [8/256/0/180] [ 877.253206] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 879.234252] LNet: 40697:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.240763] LNetError: 40697:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 879.249232] LNet: Removed LNI 192.168.202.60@tcp1 [ 879.274225] LNet: Removed LNI 10.1.2.3@tcp2 [ 879.870602] Key type .llcrypt unregistered [ 879.873870] Key type ._llcrypt unregistered [ 894.258207] Key type ._llcrypt registered [ 894.260606] Key type .llcrypt registered [ 895.648062] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 04:50:07 (1787388607) [ 895.882648] LNet: 41499:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 895.882648] mask usage: [+|-] ... [ 896.052838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 896.067482] alg: No test for adler32 (adler32-zlib) [ 897.038504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 897.094018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 897.122749] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 898.032248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 898.050507] LNet: 41664:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 898.055248] LNetError: 41664:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 898.068987] LNet: Removed LNI 192.168.202.60@tcp [ 898.081361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 898.129384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 898.156714] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 905.075138] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 04:50:17 (1787388617) [ 905.458884] LNet: 42258:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 905.478223] LNetError: 42258:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 905.499892] LNet: Removed LNI 192.168.202.60@tcp [ 906.155775] Key type .llcrypt unregistered [ 906.157516] Key type ._llcrypt unregistered [ 913.575514] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 913.858181] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 913.876650] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 913.881778] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 916.875345] Key type ._llcrypt registered [ 916.880300] Key type .llcrypt registered [ 916.949784] LNet: 42535:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 916.949784] mask usage: [+|-] ... [ 917.185768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 917.212605] alg: No test for adler32 (adler32-zlib) [ 918.206561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 918.338353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 918.369958] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 919.703520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 919.760606] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 923.071625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 923.112240] LNet: 43002:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 923.125301] LNetError: 43002:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 923.133899] LNet: Removed LNI 10.1.2.3@tcp [ 923.144353] LNet: Removed LNI 192.168.202.60@tcp [ 923.183863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 923.283642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 923.342852] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 923.349058] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 924.145463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 924.176428] LNet: 43029:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 924.187653] LNetError: 43029:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 924.200341] LNetError: 43029:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 924.212484] LNet: Removed LNI 10.1.2.3@tcp [ 924.232690] LNet: Removed LNI 192.168.202.60@tcp [ 924.256402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 924.329607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 924.373110] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 924.377667] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 925.073365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 925.112956] LNet: Added LNI 192.168.202.60@tcp2 [8/256/0/180] [ 926.401419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 926.415275] LNet: 43192:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.422630] LNetError: 43192:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 926.434143] LNetError: 43192:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 926.439868] LNet: Removed LNI 10.1.2.3@tcp [ 926.441472] LNet: Removed LNI 192.168.202.60@tcp [ 926.463894] LNet: Removed LNI 192.168.202.60@tcp2 [ 926.481666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 926.558468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 926.615051] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 926.620668] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 926.628512] LNet: Added LNI 192.168.202.60@tcp2 [8/256/0/180] [ 936.682779] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 04:50:48 (1787388648) [ 937.266849] LNet: 43789:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 937.272376] LNetError: 43789:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 937.287313] LNetError: 43789:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 937.295231] LNet: Removed LNI 10.1.2.3@tcp [ 937.300375] LNet: Removed LNI 192.168.202.60@tcp [ 937.321615] LNet: Removed LNI 192.168.202.60@tcp2 [ 938.272534] Key type .llcrypt unregistered [ 938.277873] Key type ._llcrypt unregistered [ 947.528773] Key type ._llcrypt registered [ 947.530242] Key type .llcrypt registered [ 947.577678] LNet: 44015:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 947.577678] mask usage: [+|-] ... [ 947.875099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 947.907900] alg: No test for adler32 (adler32-zlib) [ 948.945973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 949.021466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 949.049387] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 950.397690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 950.458488] LNet: 44184:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 950.466264] LNetError: 44184:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 950.512324] LNet: Removed LNI 192.168.202.60@tcp [ 950.551889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 950.642679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 950.670397] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 951.660389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 951.697663] LNet: 44214:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 951.702745] LNetError: 44214:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 951.717260] LNet: Removed LNI 192.168.202.60@tcp [ 951.735261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 951.821754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 951.862702] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 952.692213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 952.729324] LNet: 44244:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 952.732689] LNetError: 44244:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 952.742725] LNet: Removed LNI 192.168.202.60@tcp [ 952.757816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 952.806986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 952.827420] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 953.282463] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 953.440128] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 953.461578] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 953.472589] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 955.923373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 955.940775] LNet: 44317:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 955.946746] LNetError: 44317:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 955.977514] LNet: Removed LNI 192.168.202.60@tcp [ 956.006578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 956.087437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 956.106285] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 965.828888] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 04:51:17 (1787388677) [ 965.970188] LNet: 44905:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 965.993673] LNetError: 44905:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 966.016959] LNet: Removed LNI 192.168.202.60@tcp [ 967.118462] Key type .llcrypt unregistered [ 967.120065] Key type ._llcrypt unregistered [ 974.383272] Key type ._llcrypt registered [ 974.385850] Key type .llcrypt registered [ 974.464311] LNet: 45139:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 974.464311] mask usage: [+|-] ... [ 974.746638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 974.760172] alg: No test for adler32 (adler32-zlib) [ 975.699366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 975.812920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 975.858275] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 977.205731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 977.253399] LNet: 45312:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 977.269178] LNetError: 45312:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 977.283207] LNet: Removed LNI 192.168.202.60@tcp [ 977.316892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 977.389849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 977.472102] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 985.467913] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 04:51:37 (1787388697) [ 985.748159] LNet: 45904:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 985.751992] LNetError: 45904:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 985.781143] LNet: Removed LNI 192.168.202.60@tcp [ 986.527284] Key type .llcrypt unregistered [ 986.530571] Key type ._llcrypt unregistered [ 996.423329] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 996.817248] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 996.831451] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 996.833517] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 999.788940] Key type ._llcrypt registered [ 999.796668] Key type .llcrypt registered [ 999.863881] LNet: 46182:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 999.863881] mask usage: [+|-] ... [ 1000.113253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1000.141246] alg: No test for adler32 (adler32-zlib) [ 1001.070616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1001.176602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1001.239611] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1002.523891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1002.560765] LNet: 46356:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1002.567462] LNetError: 46356:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1002.588633] LNet: Removed LNI 192.168.202.60@tcp [ 1002.607765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1002.689580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1002.706991] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1010.108585] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 04:52:02 (1787388722) [ 1010.548646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1010.593367] LNet: 46955:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1010.611672] LNetError: 46955:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1010.631786] LNet: Removed LNI 192.168.202.60@tcp [ 1010.713868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1010.964958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1019.827344] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 04:52:11 (1787388731) [ 1020.121301] LNet: 47538:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1020.690560] Key type .llcrypt unregistered [ 1020.693299] Key type ._llcrypt unregistered [ 1028.045171] Key type ._llcrypt registered [ 1028.047113] Key type .llcrypt registered [ 1028.140949] LNet: 47763:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1028.140949] mask usage: [+|-] ... [ 1028.461268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1028.482883] alg: No test for adler32 (adler32-zlib) [ 1029.503630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1029.623198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1029.693469] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1030.722975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1030.736251] LNet: 47935:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1030.746601] LNetError: 47935:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1030.757869] LNet: Removed LNI 192.168.202.60@tcp [ 1030.778921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1030.866612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1030.920600] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1039.828759] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 04:52:31 (1787388751) [ 1040.314414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1040.358343] LNet: 48529:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1040.371370] LNetError: 48529:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1040.390538] LNet: Removed LNI 192.168.202.60@tcp [ 1040.423035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1040.534888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1049.048352] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 04:52:41 (1787388761) [ 1051.119779] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1053.528705] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 04:52:45 (1787388765) [ 1055.337669] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1057.092517] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 04:52:49 (1787388769) [ 1059.091254] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1061.549479] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 04:52:53 (1787388773) [ 1063.617407] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1065.674314] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 04:52:57 (1787388777) [ 1067.458340] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1069.548238] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 04:53:01 (1787388781) [ 1069.678843] LNet: 50573:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1070.634456] Key type .llcrypt unregistered [ 1070.636212] Key type ._llcrypt unregistered [ 1079.618471] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1079.825133] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1079.837199] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1079.858766] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1082.654532] Key type ._llcrypt registered [ 1082.660037] Key type .llcrypt registered [ 1082.715173] LNet: 50865:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1082.715173] mask usage: [+|-] ... [ 1082.960898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1082.988540] alg: No test for adler32 (adler32-zlib) [ 1084.071893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1084.243642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1084.279571] LNet: Added LNI 192.168.202.60@tcp [8/512/0/180] [ 1084.288542] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1084.294639] LNet: 50905:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1084.470060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1084.498242] LNet: 50924:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1084.504762] LNetError: 50924:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1084.515058] LNet: Removed LNI 10.1.2.3@tcp [ 1084.516761] LNetError: 50924:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1084.547691] LNet: Removed LNI 192.168.202.60@tcp [ 1084.572812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1084.709796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1084.745621] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1084.877584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1084.923719] LNet: 50948:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1084.934529] LNetError: 50948:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1084.965407] LNet: Removed LNI 192.168.202.60@tcp [ 1085.016772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1085.119020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1085.165845] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1085.176687] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1085.179696] LNet: 50956:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1085.372748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1085.405908] LNet: 50974:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1085.411922] LNetError: 50974:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1085.426543] LNet: Removed LNI 10.1.2.3@tcp [ 1085.429944] LNetError: 50974:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1085.453891] LNet: Removed LNI 192.168.202.60@tcp [ 1085.506369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1085.626243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1085.645757] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1085.651867] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1085.655734] LNet: 50982:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1085.876588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1085.903310] LNet: 51008:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1085.911921] LNetError: 51008:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1085.926668] LNet: Removed LNI 10.1.2.3@tcp [ 1085.929739] LNetError: 51008:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1085.941781] LNet: Removed LNI 192.168.202.60@tcp [ 1085.968644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1086.078817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1086.104035] LNet: Added LNI 192.168.202.60@tcp [8/256/0/182] [ 1086.111951] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1086.255481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1086.305746] LNet: 51036:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1086.314503] LNetError: 51036:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1086.323485] LNet: Removed LNI 10.1.2.3@tcp [ 1086.328689] LNetError: 51036:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1086.352220] LNet: Removed LNI 192.168.202.60@tcp [ 1086.373449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1086.459706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1086.497168] LNet: Added LNI 192.168.202.60@tcp [8/256/0/182] [ 1086.511463] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1086.513532] LNet: 51044:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1086.711571] LNet: Removed LNI 10.1.2.3@tcp [ 1086.731454] LNet: Removed LNI 192.168.202.60@tcp [ 1087.725744] Key type .llcrypt unregistered [ 1087.728630] Key type ._llcrypt unregistered [ 1101.035217] Key type ._llcrypt registered [ 1101.036857] Key type .llcrypt registered [ 1102.302322] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 04:53:34 (1787388814) [ 1102.610928] LNet: 51860:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1102.610928] mask usage: [+|-] ... [ 1102.742225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1102.751864] alg: No test for adler32 (adler32-zlib) [ 1103.715425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1103.762778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1103.783943] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1104.484902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.60 [ 1104.502812] LNet: 52021:0:(config.c:1627:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1104.510831] LNet: Removed LNI 192.168.202.60@tcp [ 1110.852678] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 04:53:43 (1787388823) [ 1111.222669] LNet: 52592:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1111.235084] LNetError: 52592:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1111.252335] LNet: Removed LNI 192.168.202.60@tcp [ 1111.848453] Key type .llcrypt unregistered [ 1111.850823] Key type ._llcrypt unregistered [ 1119.755312] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1120.059709] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1122.918364] Key type ._llcrypt registered [ 1122.921711] Key type .llcrypt registered [ 1122.980209] LNet: 52875:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1122.980209] mask usage: [+|-] ... [ 1123.164752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1123.175320] alg: No test for adler32 (adler32-zlib) [ 1124.520136] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1131.276994] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 04:54:03 (1787388843) [ 1131.564932] LNet: 53507:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1131.567871] LNetError: 53507:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1131.580661] LNet: Removed LNI 10.1.2.3@tcp [ 1132.109985] Key type .llcrypt unregistered [ 1132.112830] Key type ._llcrypt unregistered [ 1138.629529] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1138.901822] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1138.914330] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1138.923901] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1140.519480] Key type ._llcrypt registered [ 1140.521755] Key type .llcrypt registered [ 1140.544700] LNet: 53772:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1140.544700] mask usage: [+|-] ... [ 1140.689596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1140.707811] alg: No test for adler32 (adler32-zlib) [ 1141.736483] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1148.430947] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 04:54:20 (1787388860) [ 1150.330242] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1152.132425] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 04:54:24 (1787388864) [ 1152.226643] LNet: 54675:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1152.243118] LNetError: 54675:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1152.278306] LNet: Removed LNI 10.1.2.3@tcp [ 1152.814563] Key type .llcrypt unregistered [ 1152.817971] Key type ._llcrypt unregistered [ 1159.466632] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1159.728835] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1159.734793] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1159.745971] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1162.731415] Key type ._llcrypt registered [ 1162.739322] Key type .llcrypt registered [ 1162.792316] LNet: 54955:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1162.792316] mask usage: [+|-] ... [ 1163.027948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1163.047299] alg: No test for adler32 (adler32-zlib) [ 1164.015477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1164.085674] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1173.816245] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 04:54:45 (1787388885) [ 1174.633752] LNet: 55911:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1174.643801] LNetError: 55911:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1174.662350] LNet: Removed LNI 10.1.2.3@tcp [ 1175.590181] Key type .llcrypt unregistered [ 1175.592362] Key type ._llcrypt unregistered [ 1182.897320] Key type ._llcrypt registered [ 1182.899205] Key type .llcrypt registered [ 1182.931740] LNet: 56135:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1182.931740] mask usage: [+|-] ... [ 1183.206510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1183.214876] alg: No test for adler32 (adler32-zlib) [ 1184.049720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1184.065541] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1192.838665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.160@tcp [ 1193.469473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp -m GET -e local_interrupt -r 1 [ 1193.508367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp -m GET -e local_interrupt -r 1 [ 1193.537230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.160@tcp [ 1193.554623] LNet: There was an unexpected network error while writing to 192.168.202.160: rc = -22 [ 1193.765484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1193.789540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1193.824162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1193.862885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1193.933085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1193.988925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1194.049549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1194.093850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1194.127920] LNet: 56350:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1194.164443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1194.222186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1194.315384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1194.344581] LNet: 1 local NIs in recovery (showing 1): 192.168.202.60@tcp [ 1194.364163] LNet: There was an unexpected network error while writing to 192.168.202.60: rc = -22 [ 1194.406110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1194.463268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1194.562146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1194.649986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1195.522399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp -m GET -e local_dropped -r 1 [ 1195.621531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp -m GET -e local_dropped -r 1 [ 1195.725953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.160@tcp [ 1195.764799] LNet: There was an unexpected network error while writing to 192.168.202.160: rc = -22 [ 1196.107336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1196.187016] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1196.291188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1196.368308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1196.425229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1196.479949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1196.546038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1196.614424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1196.649528] LNet: 56459:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1196.697047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1196.758510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1196.793261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1196.835896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1196.875198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1196.924330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1196.963950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1197.623537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp -m GET -e local_aborted -r 1 [ 1197.711144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp -m GET -e local_aborted -r 1 [ 1197.779388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.160@tcp [ 1197.816662] LNet: There was an unexpected network error while writing to 192.168.202.160: rc = -22 [ 1198.165556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1198.229023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1198.289182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1198.363939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1198.406248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1198.472577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1198.546282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1198.607290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1198.626403] LNet: 56569:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1198.645901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1198.691983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1198.743240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1198.792258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1198.889129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1198.944076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1198.989225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1199.708861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp -m GET -e local_no_route -r 1 [ 1199.781791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp -m GET -e local_no_route -r 1 [ 1199.844703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.160@tcp [ 1200.242201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1200.271946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1200.295964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1200.315778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1200.335162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1200.357443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1200.389193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1200.436919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1200.515434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1200.551607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1200.585425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1200.619314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1200.643404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1200.669426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1200.699173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1201.402846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp -m GET -e local_timeout -r 1 [ 1201.444921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp -m GET -e local_timeout -r 1 [ 1201.495891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.160@tcp [ 1201.709768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1201.763588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1201.821905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1201.877565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1201.939615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1202.020433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1202.103487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1202.155730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1202.184828] LNet: 56788:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1202.193821] LNet: 56788:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1202.221768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1202.279535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1202.344805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1202.367707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1202.392307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1202.423308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1202.475814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1203.202919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp -m GET -e local_error -r 1 [ 1203.241829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp -m GET -e local_error -r 1 [ 1203.264844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.160@tcp [ 1203.283418] LNet: There was an unexpected network error while writing to 192.168.202.160: rc = -22 [ 1203.288292] LNet: Skipped 2 previous similar messages [ 1203.473676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1203.501477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1203.530857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1203.558419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1203.581489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1203.608346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1203.661912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1203.708900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1203.775147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1203.852473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1203.906829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1203.987610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1204.039956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1204.102986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1204.166680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1204.656781] LNet: 56938:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1204.684184] LNetError: 56938:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1204.710144] LNet: Removed LNI 192.168.202.60@tcp [ 1205.359565] Key type .llcrypt unregistered [ 1205.370805] Key type ._llcrypt unregistered [ 1226.009045] Key type ._llcrypt registered [ 1226.010940] Key type .llcrypt registered [ 1227.232863] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 04:55:39 (1787388939) [ 1229.043388] Key type .llcrypt unregistered [ 1229.046911] Key type ._llcrypt unregistered [ 1237.470275] Key type ._llcrypt registered [ 1237.474190] Key type .llcrypt registered [ 1237.555332] LNet: 57978:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1237.555332] mask usage: [+|-] ... [ 1237.875598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1237.917162] alg: No test for adler32 (adler32-zlib) [ 1238.804119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1238.822082] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1247.983471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.160@tcp [ 1258.547495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1258.621348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1258.663456] LNet: Added LNI 192.168.202.60@tcp1 [8/256/0/180] [ 1260.775618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp -m GET -e local_interrupt -r 1 [ 1260.816655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp1 -m GET -e local_interrupt -r 1 [ 1260.898272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp -m GET -e local_interrupt -r 1 [ 1260.953123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp1 -m GET -e local_interrupt -r 1 [ 1261.028953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp1 -d 192.168.202.160@tcp -m GET -e local_interrupt -r 1 [ 1261.101467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp1 -d 192.168.202.160@tcp1 -m GET -e local_interrupt -r 1 [ 1261.159233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp1 -d 192.168.202.60@tcp -m GET -e local_interrupt -r 1 [ 1261.244116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp1 -d 192.168.202.60@tcp1 -m GET -e local_interrupt -r 1 [ 1261.294735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.160@tcp [ 1261.311444] LNet: There was an unexpected network error while writing to 192.168.202.160: rc = -22 [ 1261.320494] LNet: 1 local NIs in recovery (showing 1): 192.168.202.60@tcp [ 1261.657673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1261.686566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1261.726382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1261.797208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1261.895049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1262.023951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1262.122583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1262.199804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1262.235318] LNet: 58409:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1262.278779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1262.349611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1262.412537] LNet: There was an unexpected network error while writing to 192.168.202.60: rc = -22 [ 1262.443109] LNet: Skipped 3 previous similar messages [ 1262.469240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1262.516170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1262.551061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1262.591370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1262.660443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1263.718264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp -m GET -e local_dropped -r 1 [ 1263.782556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp1 -m GET -e local_dropped -r 1 [ 1263.839072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp -m GET -e local_dropped -r 1 [ 1263.896104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp1 -m GET -e local_dropped -r 1 [ 1263.933806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp1 -d 192.168.202.160@tcp -m GET -e local_dropped -r 1 [ 1263.959923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp1 -d 192.168.202.160@tcp1 -m GET -e local_dropped -r 1 [ 1263.994562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp1 -d 192.168.202.60@tcp -m GET -e local_dropped -r 1 [ 1264.028498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp1 -d 192.168.202.60@tcp1 -m GET -e local_dropped -r 1 [ 1264.056198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.160@tcp [ 1264.083449] LNet: There was an unexpected network error while writing to 192.168.202.160: rc = -22 [ 1264.226822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1264.269922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1264.297468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1264.321642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1264.356852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1264.398576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1264.428794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1264.470568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1264.507411] LNet: 58531:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1264.545145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1264.596953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1264.659168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1264.736252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1264.774407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1264.809335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1264.874153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1265.552206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp -m GET -e local_aborted -r 1 [ 1265.583366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp1 -m GET -e local_aborted -r 1 [ 1265.619932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp -m GET -e local_aborted -r 1 [ 1265.653812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp1 -m GET -e local_aborted -r 1 [ 1265.683161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp1 -d 192.168.202.160@tcp -m GET -e local_aborted -r 1 [ 1265.721781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp1 -d 192.168.202.160@tcp1 -m GET -e local_aborted -r 1 [ 1265.791255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp1 -d 192.168.202.60@tcp -m GET -e local_aborted -r 1 [ 1265.863929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp1 -d 192.168.202.60@tcp1 -m GET -e local_aborted -r 1 [ 1265.942221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.160@tcp [ 1266.293494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1266.326593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1266.357835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1266.381509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1266.411584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1266.446700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1266.473970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1266.514373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1266.553390] LNet: 58652:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1266.581419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1266.640401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1266.692601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1266.743826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1266.814907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1266.862451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1266.910700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1267.670471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp -m GET -e local_no_route -r 1 [ 1267.718289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp1 -m GET -e local_no_route -r 1 [ 1267.754326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp -m GET -e local_no_route -r 1 [ 1267.783440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp1 -m GET -e local_no_route -r 1 [ 1267.825345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp1 -d 192.168.202.160@tcp -m GET -e local_no_route -r 1 [ 1267.857492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp1 -d 192.168.202.160@tcp1 -m GET -e local_no_route -r 1 [ 1267.888444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp1 -d 192.168.202.60@tcp -m GET -e local_no_route -r 1 [ 1267.916449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp1 -d 192.168.202.60@tcp1 -m GET -e local_no_route -r 1 [ 1267.949721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.160@tcp [ 1267.968279] LNet: There was an unexpected network error while writing to 192.168.202.160: rc = -22 [ 1267.974464] LNet: Skipped 5 previous similar messages [ 1268.280381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1268.326220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1268.377156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1268.416273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1268.479486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1268.520177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1268.553965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1268.593232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1268.614964] LNet: 58773:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1268.634266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1268.668456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1268.698989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1268.733330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1268.789499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1268.868385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1268.941072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1269.654062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp -m GET -e local_timeout -r 1 [ 1269.684344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp1 -m GET -e local_timeout -r 1 [ 1269.729285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp -m GET -e local_timeout -r 1 [ 1269.777898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp1 -m GET -e local_timeout -r 1 [ 1269.834096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp1 -d 192.168.202.160@tcp -m GET -e local_timeout -r 1 [ 1269.884782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp1 -d 192.168.202.160@tcp1 -m GET -e local_timeout -r 1 [ 1269.956390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp1 -d 192.168.202.60@tcp -m GET -e local_timeout -r 1 [ 1270.022461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp1 -d 192.168.202.60@tcp1 -m GET -e local_timeout -r 1 [ 1270.053893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.160@tcp [ 1270.298397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1270.332733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1270.365601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1270.413420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1270.457859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1270.481176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1270.532548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1270.570792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1270.592932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1270.620818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1270.641669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1270.665986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1270.688854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1270.715455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1270.741334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1271.134447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp -m GET -e local_error -r 1 [ 1271.165769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp1 -m GET -e local_error -r 1 [ 1271.195982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp -m GET -e local_error -r 1 [ 1271.226464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp1 -m GET -e local_error -r 1 [ 1271.257186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp1 -d 192.168.202.160@tcp -m GET -e local_error -r 1 [ 1271.287207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp1 -d 192.168.202.160@tcp1 -m GET -e local_error -r 1 [ 1271.315679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp1 -d 192.168.202.60@tcp -m GET -e local_error -r 1 [ 1271.345135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp1 -d 192.168.202.60@tcp1 -m GET -e local_error -r 1 [ 1271.376956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.160@tcp [ 1271.572519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1271.603549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1271.639310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1271.666848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1271.688540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1271.713893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1271.743719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1271.767907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1271.794829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1271.826777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1271.856760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1271.895888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1271.922421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1271.974280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1272.018804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1278.258674] LNet: 59092:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1278.263969] LNetError: 59092:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1279.327973] LNet: Removed LNI 192.168.202.60@tcp [ 1279.330877] LNetError: 59092:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1279.350478] LNet: Removed LNI 192.168.202.60@tcp1 [ 1279.766544] Key type .llcrypt unregistered [ 1279.770736] Key type ._llcrypt unregistered [ 1298.148134] Key type ._llcrypt registered [ 1298.152639] Key type .llcrypt registered [ 1299.428298] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 04:56:51 (1787389011) [ 1300.345846] Key type .llcrypt unregistered [ 1300.348349] Key type ._llcrypt unregistered [ 1307.865635] Key type ._llcrypt registered [ 1307.871250] Key type .llcrypt registered [ 1307.929555] LNet: 60134:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1307.929555] mask usage: [+|-] ... [ 1308.248363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1308.286122] alg: No test for adler32 (adler32-zlib) [ 1309.167232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1309.188079] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1317.689756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.160@tcp [ 1318.537450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp -m GET -e remote_dropped -r 1 [ 1318.576381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp -m GET -e remote_dropped -r 1 [ 1318.611425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.160@tcp [ 1318.659393] LNet: There was an unexpected network error while writing to 192.168.202.160: rc = -22 [ 1318.988757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1319.034600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1319.118601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1319.183238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1319.232452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1319.273560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1319.315924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1319.368726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1319.394526] LNet: 60349:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1319.424088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1319.507300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1319.566847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1319.624863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1319.690846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1319.731070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1319.805896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1320.387747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp -m GET -e remote_error -r 1 [ 1320.424964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp -m GET -e remote_error -r 1 [ 1320.456788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.160@tcp [ 1320.472539] LNet: There was an unexpected network error while writing to 192.168.202.160: rc = -22 [ 1320.711681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1320.767648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1320.838496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1320.892640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1320.941048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1320.974818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1321.001423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1321.028773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1321.045746] LNet: 60458:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1321.071845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1321.111387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1321.142394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1321.169334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1321.198659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1321.227205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1321.256585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1321.929209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp -m GET -e remote_timeout -r 1 [ 1321.981055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp -m GET -e remote_timeout -r 1 [ 1322.053339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.160@tcp [ 1322.087093] LNet: There was an unexpected network error while writing to 192.168.202.160: rc = -22 [ 1322.324691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1322.366397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1322.395869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1322.425376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1322.453853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1322.481389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1322.509878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1322.537164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1322.550891] LNet: 60567:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1322.568484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1322.590381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1322.632799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1322.660441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1322.689625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1322.731001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1322.799715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1323.228957] LNet: 60608:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1323.242238] LNetError: 60608:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1323.259323] LNet: Removed LNI 192.168.202.60@tcp [ 1323.790399] Key type .llcrypt unregistered [ 1323.792786] Key type ._llcrypt unregistered [ 1343.708890] Key type ._llcrypt registered [ 1343.715130] Key type .llcrypt registered [ 1345.283658] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 04:57:37 (1787389057) [ 1346.531371] Key type .llcrypt unregistered [ 1346.535348] Key type ._llcrypt unregistered [ 1353.391758] Key type ._llcrypt registered [ 1353.399413] Key type .llcrypt registered [ 1353.475981] LNet: 61647:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1353.475981] mask usage: [+|-] ... [ 1353.671748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1353.685434] alg: No test for adler32 (adler32-zlib) [ 1354.692718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1354.731380] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1363.468927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.160@tcp [ 1374.304854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1374.352378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1374.380524] LNet: Added LNI 192.168.202.60@tcp1 [8/256/0/180] [ 1376.507261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp -m GET -e remote_dropped -r 1 [ 1376.551842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp1 -m GET -e remote_dropped -r 1 [ 1376.621787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp -m GET -e remote_dropped -r 1 [ 1376.683317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp1 -m GET -e remote_dropped -r 1 [ 1376.738760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp1 -d 192.168.202.160@tcp -m GET -e remote_dropped -r 1 [ 1376.789792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp1 -d 192.168.202.160@tcp1 -m GET -e remote_dropped -r 1 [ 1376.845403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp1 -d 192.168.202.60@tcp -m GET -e remote_dropped -r 1 [ 1376.889037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp1 -d 192.168.202.60@tcp1 -m GET -e remote_dropped -r 1 [ 1376.948683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.160@tcp [ 1376.978389] LNet: There was an unexpected network error while writing to 192.168.202.160: rc = -22 [ 1376.983591] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.160@tcp [ 1377.244589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1377.294094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1377.340916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1377.396973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1377.453920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1377.499906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1377.552431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1377.608371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1377.627940] LNet: 62076:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1377.658213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1377.710727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1377.744287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1377.781757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1377.813151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1377.841741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1377.888248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1377.918941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1378.527102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp -m GET -e remote_error -r 1 [ 1378.559275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp1 -m GET -e remote_error -r 1 [ 1378.606067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp -m GET -e remote_error -r 1 [ 1378.646759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp1 -m GET -e remote_error -r 1 [ 1378.678914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp1 -d 192.168.202.160@tcp -m GET -e remote_error -r 1 [ 1378.710377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp1 -d 192.168.202.160@tcp1 -m GET -e remote_error -r 1 [ 1378.746799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp1 -d 192.168.202.60@tcp -m GET -e remote_error -r 1 [ 1378.791970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp1 -d 192.168.202.60@tcp1 -m GET -e remote_error -r 1 [ 1378.842871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.160@tcp [ 1378.878278] LNet: There was an unexpected network error while writing to 192.168.202.160: rc = -22 [ 1378.888135] LNet: Skipped 2 previous similar messages [ 1379.150535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1379.214739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1379.257765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1379.292324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1379.335335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1379.373505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1379.422186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1379.469388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1379.494082] LNet: 62200:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1379.532239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1379.578306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1379.652073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1379.682756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1379.711333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1379.748178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1379.824603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1379.927251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1380.822343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp -m GET -e remote_timeout -r 1 [ 1380.883434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp1 -m GET -e remote_timeout -r 1 [ 1380.948344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp -m GET -e remote_timeout -r 1 [ 1380.983044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp1 -m GET -e remote_timeout -r 1 [ 1381.020994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp1 -d 192.168.202.160@tcp -m GET -e remote_timeout -r 1 [ 1381.058815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp1 -d 192.168.202.160@tcp1 -m GET -e remote_timeout -r 1 [ 1381.086976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp1 -d 192.168.202.60@tcp -m GET -e remote_timeout -r 1 [ 1381.124666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp1 -d 192.168.202.60@tcp1 -m GET -e remote_timeout -r 1 [ 1381.164753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.160@tcp [ 1381.181280] LNet: There was an unexpected network error while writing to 192.168.202.160: rc = -22 [ 1381.186287] LNet: Skipped 1 previous similar message [ 1381.434667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1381.470950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1381.501541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1381.529986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1381.561362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1381.584909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1381.612336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1381.638480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1381.652873] LNet: 62323:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1381.667860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1381.688698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1381.713463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1381.739454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1381.765466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1381.797081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1381.824486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1381.850312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1387.561381] LNet: 62401:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1387.567792] LNetError: 62401:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1387.583365] LNet: Removed LNI 192.168.202.60@tcp [ 1387.595937] LNet: Removed LNI 192.168.202.60@tcp1 [ 1388.185456] Key type .llcrypt unregistered [ 1388.187637] Key type ._llcrypt unregistered [ 1408.425925] Key type ._llcrypt registered [ 1408.428946] Key type .llcrypt registered [ 1409.656839] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 04:58:41 (1787389121) [ 1411.354983] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1413.228128] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 04:58:45 (1787389125) [ 1414.167464] Key type .llcrypt unregistered [ 1414.170230] Key type ._llcrypt unregistered [ 1422.290895] Key type ._llcrypt registered [ 1422.297463] Key type .llcrypt registered [ 1422.363794] LNet: 63731:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1422.363794] mask usage: [+|-] ... [ 1422.717448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1422.737281] alg: No test for adler32 (adler32-zlib) [ 1423.855889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1423.905635] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1432.213550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.160@tcp [ 1433.120696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp -m GET -e network_timeout -r 1 [ 1433.171074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp -m GET -e network_timeout -r 1 [ 1433.219870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.160@tcp [ 1433.259280] LNet: There was an unexpected network error while writing to 192.168.202.160: rc = -22 [ 1433.522423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1433.544473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1433.568242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1433.594565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1433.638590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1433.680660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1433.726500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1433.793418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1433.827873] LNet: 63946:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1433.864437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1433.898790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1433.938154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1433.989154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1434.019624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1434.049640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1434.297655] LNet: 63985:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1434.303734] LNetError: 63985:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1434.317784] LNet: Removed LNI 192.168.202.60@tcp [ 1434.728564] Key type .llcrypt unregistered [ 1434.731978] Key type ._llcrypt unregistered [ 1457.195945] Key type ._llcrypt registered [ 1457.201636] Key type .llcrypt registered [ 1457.272553] LNet: 64434:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1457.272553] mask usage: [+|-] ... [ 1457.614566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1457.641138] alg: No test for adler32 (adler32-zlib) [ 1458.606250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1458.630521] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1468.408206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.160@tcp [ 1479.248984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1479.311667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1479.347246] LNet: Added LNI 192.168.202.60@tcp1 [8/256/0/180] [ 1481.357781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp -m GET -e network_timeout -r 1 [ 1481.390316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp1 -m GET -e network_timeout -r 1 [ 1481.427991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp -m GET -e network_timeout -r 1 [ 1481.470895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp1 -m GET -e network_timeout -r 1 [ 1481.568449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp1 -d 192.168.202.160@tcp -m GET -e network_timeout -r 1 [ 1481.603839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp1 -d 192.168.202.160@tcp1 -m GET -e network_timeout -r 1 [ 1481.644655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp1 -d 192.168.202.60@tcp -m GET -e network_timeout -r 1 [ 1481.688954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp1 -d 192.168.202.60@tcp1 -m GET -e network_timeout -r 1 [ 1481.725541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.160@tcp [ 1481.743168] LNet: There was an unexpected network error while writing to 192.168.202.160: rc = -22 [ 1482.045092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1482.119437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1482.183994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1482.209416] LNet: 1 local NIs in recovery (showing 1): 192.168.202.60@tcp [ 1482.222292] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.160@tcp [ 1482.233287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1482.301854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1482.374518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1482.436075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1482.501944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1482.530714] LNet: 64864:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1482.566453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1482.609613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1482.676741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1482.745597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1482.778952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1482.846223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1488.408815] LNet: There was an unexpected network error while writing to 192.168.202.160: rc = -22 [ 1488.432147] LNet: Skipped 2 previous similar messages [ 1488.867901] LNet: 64938:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1488.886627] LNetError: 64938:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1489.887243] LNet: Removed LNI 192.168.202.60@tcp [ 1489.894107] LNetError: 64938:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1489.920789] LNet: Removed LNI 192.168.202.60@tcp1 [ 1490.595190] Key type .llcrypt unregistered [ 1490.596754] Key type ._llcrypt unregistered [ 1510.786643] Key type ._llcrypt registered [ 1510.793634] Key type .llcrypt registered [ 1512.134344] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 05:00:24 (1787389224) [ 1512.538663] LNet: 65739:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1512.538663] mask usage: [+|-] ... [ 1512.853086] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1512.897969] alg: No test for adler32 (adler32-zlib) [ 1513.819332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1513.876757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1513.900218] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1514.901428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1514.964805] LNet: Added LNI 192.168.202.60@tcp1 [8/256/0/180] [ 1516.146734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.60@tcp [ 1516.278383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1516.354929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1516.435115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1516.501202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1516.581274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1516.626462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.60@tcp [ 1516.657191] LNetError: 66038:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.202.60@tcp to recovery [ 1516.895440] LNet: There was an unexpected network error while writing to 192.168.202.60: rc = -22 [ 1516.895305] LNet: 1 local NIs in recovery (showing 1): 192.168.202.60@tcp [ 1518.944130] LNet: There was an unexpected network error while writing to 192.168.202.60: rc = -22 [ 1519.245259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1522.016705] LNet: There was an unexpected network error while writing to 192.168.202.60: rc = -22 [ 1522.996931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1523.054839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1523.142361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1523.184381] LNet: 66135:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1523.193814] LNetError: 66135:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1523.201432] LNet: Removed LNI 192.168.202.60@tcp [ 1523.218591] LNet: Removed LNI 192.168.202.60@tcp1 [ 1523.265351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1523.332060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1523.388068] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1524.651526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1524.706460] LNet: Added LNI 192.168.202.60@tcp1 [8/256/0/180] [ 1525.998505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.60@tcp [ 1526.034468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1526.112700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1526.164627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1526.199440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1526.247722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1526.285667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1526.359986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.60@tcp [ 1526.414708] LNetError: 66408:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.202.60@tcp to recovery [ 1527.391595] LNet: There was an unexpected network error while writing to 192.168.202.60: rc = -22 [ 1530.163734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1533.537466] LNet: There was an unexpected network error while writing to 192.168.202.60: rc = -22 [ 1533.562503] LNet: Skipped 1 previous similar message [ 1538.555521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1538.607840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1538.636050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1545.273788] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 05:00:57 (1787389257) [ 1545.528448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1545.561235] LNet: 67125:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1545.574582] LNetError: 67125:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1545.583835] LNetError: 67125:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1545.601654] LNet: Removed LNI 192.168.202.60@tcp [ 1545.621281] LNet: Removed LNI 192.168.202.60@tcp1 [ 1545.633807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1545.683428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1545.731134] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1546.558915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1546.578053] LNet: Added LNI 192.168.202.60@tcp1 [8/256/0/180] [ 1547.420174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.60@tcp [ 1547.526447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1547.568391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1547.609710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1547.665441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1547.696591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1547.727597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.60@tcp --health 0 [ 1547.811472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1548.706776] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.60@tcp [ 1548.709763] LNet: There was an unexpected network error while writing to 192.168.202.60: rc = -22 [ 1548.735418] LNet: Skipped 1 previous similar message [ 1562.664216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1562.961254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1563.049589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1563.094750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.60@tcp --health 500 [ 1563.318446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1569.261043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1569.369247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1569.398424] LNet: 67614:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1569.404726] LNetError: 67614:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1569.413623] LNetError: 67614:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1569.420534] LNet: Removed LNI 192.168.202.60@tcp [ 1569.430523] LNet: Removed LNI 192.168.202.60@tcp1 [ 1569.469150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1569.576276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1569.642147] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1570.693057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1570.739603] LNet: Added LNI 192.168.202.60@tcp1 [8/256/0/180] [ 1572.110342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.60@tcp [ 1572.169544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1572.270294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1572.329771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1572.392890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1572.450371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1572.505777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1572.566735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.60@tcp --health 0 [ 1573.663769] LNet: There was an unexpected network error while writing to 192.168.202.60: rc = -22 [ 1573.669154] LNet: Skipped 3 previous similar messages [ 1574.014931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1584.806834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1584.861873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1584.907991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1593.774732] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 05:01:45 (1787389305) [ 1594.270590] LNet: 68661:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1594.276099] LNetError: 68661:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1594.285717] LNetError: 68661:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1594.291104] LNet: Removed LNI 192.168.202.60@tcp [ 1594.319659] LNet: Removed LNI 192.168.202.60@tcp1 [ 1595.056521] Key type .llcrypt unregistered [ 1595.065717] Key type ._llcrypt unregistered [ 1603.055530] Key type ._llcrypt registered [ 1603.057271] Key type .llcrypt registered [ 1603.132614] LNet: 68884:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1603.132614] mask usage: [+|-] ... [ 1603.508887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1603.545257] alg: No test for adler32 (adler32-zlib) [ 1604.547725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1604.600550] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1615.855933] Lustre: DEBUG MARKER: Initial discovery [ 1615.898534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.160@tcp [ 1618.843571] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1618.974658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1619.037867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.160@tcp [ 1619.078993] LNet: There was an unexpected network error while writing to 192.168.202.160: rc = -22 [ 1619.117434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1619.182322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.160@tcp -r 1 -m GET -l 3 [ 1619.226822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.160@tcp -r 1 -m GET -e local_error [ 1619.255818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.160@tcp -r 1 -m PUT -l 6 [ 1620.064252] LNet: 1 local NIs in recovery (showing 1): 192.168.202.60@tcp [ 1620.760532] Lustre: DEBUG MARKER: Force oleg260-client.virtnet to discover 192.168.202.160@tcp (in background) [ 1621.240524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.160@tcp [ 1623.097817] Lustre: DEBUG MARKER: Force oleg260-server to discover 192.168.202.60@tcp [ 1624.363916] LNet: There was an unexpected network error while writing to 192.168.202.160: rc = -22 [ 1626.365799] Lustre: DEBUG MARKER: Wait for 69425 [ 1628.045587] Lustre: DEBUG MARKER: Finished wait on 69425 [ 1628.150692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1628.213342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1628.215364] LNet: There was an unexpected network error while writing to 192.168.202.160: rc = -22 [ 1628.509986] LNet: 69874:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1628.516632] LNetError: 69874:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1628.535569] LNet: Removed LNI 192.168.202.60@tcp [ 1629.086317] Key type .llcrypt unregistered [ 1629.087995] Key type ._llcrypt unregistered [ 1648.273933] Key type ._llcrypt registered [ 1648.277563] Key type .llcrypt registered [ 1649.764880] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 05:02:41 (1787389361) [ 1650.331590] Key type .llcrypt unregistered [ 1650.334887] Key type ._llcrypt unregistered [ 1656.943877] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1657.057222] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1657.066703] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1657.074961] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1659.722464] Key type ._llcrypt registered [ 1659.725186] Key type .llcrypt registered [ 1659.765779] LNet: 70916:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1659.765779] mask usage: [+|-] ... [ 1659.946724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1659.970240] alg: No test for adler32 (adler32-zlib) [ 1660.898476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1660.966893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1660.995478] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1661.730222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1661.754932] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1670.776248] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 05:03:03 (1787389383) [ 1671.110133] LNet: 71950:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1671.118316] LNetError: 71950:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1671.124872] LNet: Removed LNI 10.1.2.3@tcp [ 1671.133181] LNet: Removed LNI 192.168.202.60@tcp [ 1671.860271] Key type .llcrypt unregistered [ 1671.863651] Key type ._llcrypt unregistered [ 1679.948920] Key type ._llcrypt registered [ 1679.950884] Key type .llcrypt registered [ 1679.987618] LNet: 72200:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1679.987618] mask usage: [+|-] ... [ 1680.190533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1680.209689] alg: No test for adler32 (adler32-zlib) [ 1681.219981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1681.281352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1681.305254] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1682.254361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1682.296495] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1683.296873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.60@tcp [ 1684.383284] LNetError: Unexpected error -22 connecting to 192.168.202.60@tcp at host 192.168.202.60:988 [ 1690.095312] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 05:03:22 (1787389402) [ 1690.244365] LNet: 73077:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1690.256518] LNetError: 73077:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1691.295362] LNet: Removed LNI 10.1.2.3@tcp [ 1691.298279] LNetError: 73077:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1691.337200] LNet: Removed LNI 192.168.202.60@tcp [ 1692.475483] Key type .llcrypt unregistered [ 1692.479872] Key type ._llcrypt unregistered [ 1700.051122] Key type ._llcrypt registered [ 1700.054481] Key type .llcrypt registered [ 1700.128438] LNet: 73314:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1700.128438] mask usage: [+|-] ... [ 1700.364422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1700.384691] alg: No test for adler32 (adler32-zlib) [ 1701.344608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1701.389543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1701.412381] LNet: Added LNI 192.168.202.60@tcp1 [8/256/0/180] [ 1702.135151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1702.149280] LNet: Added LNI 192.168.202.60@tcp2 [8/256/0/180] [ 1703.190863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.60@tcp1 --nid 192.168.202.60@tcp2 [ 1707.782699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.60@tcp1 192.168.202.60@tcp2 [ 1707.805001] LNetError: 73918:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.60@tcp1 to 192.168.202.60@tcp2 - Route table may be misconfigured [ 1707.832729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.60@tcp2 192.168.202.60@tcp1 [ 1714.360501] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 05:03:46 (1787389426) [ 1714.576229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1714.643925] LNet: 74492:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1714.652216] LNetError: 74492:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1714.659387] LNet: Removed LNI 192.168.202.60@tcp1 [ 1715.686693] LNet: Removed LNI 192.168.202.60@tcp2 [ 1715.740484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1715.798816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1715.817416] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1716.499883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1716.514631] LNet: Added LNI 192.168.202.60@tcp1 [8/256/0/180] [ 1717.213709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.60@tcp [ 1717.251268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.60@tcp 192.168.202.60@tcp [ 1717.278171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.60@tcp1 192.168.202.60@tcp1 [ 1717.307063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.60@tcp -d 192.168.202.60@tcp -e network_timeout [ 1717.333807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.60@tcp -d 192.168.202.60@tcp1 -e network_timeout [ 1717.366774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.60@tcp1 -d 192.168.202.60@tcp -e network_timeout [ 1717.401696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.60@tcp1 -d 192.168.202.60@tcp1 -e network_timeout [ 1717.438868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.60@tcp [ 1717.461286] LNet: There was an unexpected network error while writing to 192.168.202.60: rc = -22 [ 1717.479908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1717.564961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1717.670647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1717.859376] LNet: 1 local NIs in recovery (showing 1): 192.168.202.60@tcp [ 1723.206438] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 05:03:55 (1787389435) [ 1723.309939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1723.347961] LNet: 75350:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1723.361674] LNetError: 75350:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1723.371118] LNetError: 75350:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1723.376718] LNet: Removed LNI 192.168.202.60@tcp [ 1724.391965] LNet: Removed LNI 192.168.202.60@tcp1 [ 1724.482319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1724.608226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1724.899630] LNet: 75391:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1725.237519] Key type .llcrypt unregistered [ 1725.239366] Key type ._llcrypt unregistered [ 1737.339676] Key type ._llcrypt registered [ 1737.341732] Key type .llcrypt registered [ 1738.726863] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 05:04:10 (1787389450) [ 1739.606122] LNet: 76163:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1739.606122] mask usage: [+|-] ... [ 1739.868438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1739.881757] alg: No test for adler32 (adler32-zlib) [ 1740.867746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1742.168347] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1744.497098] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 05:04:16 (1787389456) [ 1744.576513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1744.629136] LNet: 76484:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1744.678686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1744.743765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1744.766394] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1745.515155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1745.536509] LNet: Added LNI 192.168.202.60@tcp1 [8/256/0/180] [ 1746.528713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.60@tcp [ 1746.588311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.60@tcp1 [ 1746.633164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.60@tcp1 [ 1752.945722] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 05:04:25 (1787389465) [ 1754.865395] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg260-server [ 1756.855356] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 05:04:28 (1787389468) [ 1759.385857] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg260-server [ 1761.766669] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 05:04:33 (1787389473) [ 1764.044568] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg260-server [ 1766.056982] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 05:04:38 (1787389478) [ 1768.203865] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg260-server [ 1770.107268] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 05:04:42 (1787389482) [ 1772.235278] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg260-server [ 1774.934770] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 05:04:46 (1787389486) [ 1777.174771] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg260-server [ 1778.974068] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 05:04:51 (1787389491) [ 1781.330509] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg260-server [ 1783.421230] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 05:04:55 (1787389495) [ 1786.110501] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg260-server [ 1788.860759] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 05:05:00 (1787389500) [ 1791.564347] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg260-server [ 1793.641810] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 05:05:05 (1787389505) [ 1793.867208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1793.893515] LNet: 80365:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1793.912986] LNetError: 80365:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1794.978315] LNet: Removed LNI 192.168.202.60@tcp [ 1794.993737] LNetError: 80365:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1795.037034] LNet: Removed LNI 192.168.202.60@tcp1 [ 1795.098226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1795.173241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1795.209698] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1796.014905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1796.104552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.60@tcp [ 1796.381954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1796.418991] LNet: 80518:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1796.428684] LNetError: 80518:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1796.443848] LNet: Removed LNI 192.168.202.60@tcp [ 1796.480284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1796.569179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1796.588720] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1797.325872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.60@tcp [ 1797.507851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1798.569235] LNet: Removed LNI 192.168.202.60@tcp [ 1798.598611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1798.693243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1798.716886] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1799.520356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1799.591933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.60@tcp [ 1799.834455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1799.863513] LNet: 80811:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1799.874549] LNet: 80811:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1799.891665] LNetError: 80811:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1799.903223] LNetError: 80811:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1799.922350] LNet: Removed LNI 192.168.202.60@tcp [ 1799.957617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1800.062718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1800.105729] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1800.921266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.60@tcp [ 1801.174563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1802.215509] LNet: Removed LNI 192.168.202.60@tcp [ 1802.291431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1802.393269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1802.435096] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1803.366295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1803.483183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.60@tcp [ 1803.652124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1804.713894] LNet: Removed LNI 192.168.202.60@tcp [ 1804.749678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1804.832235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1804.855401] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1805.881255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.60@tcp [ 1806.148552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1806.204443] LNet: 81249:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1806.211977] LNet: 81249:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1806.250960] LNetError: 81249:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1806.262611] LNetError: 81249:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1806.283688] LNet: Removed LNI 192.168.202.60@tcp [ 1806.329438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1806.388927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1806.438357] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1807.295446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1807.372454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.60@tcp [ 1807.534971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1808.613680] LNet: Removed LNI 192.168.202.60@tcp [ 1808.671465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1808.746764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1808.803841] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1809.676302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.60@tcp [ 1809.948076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1811.055142] LNet: Removed LNI 192.168.202.60@tcp [ 1811.090572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1811.217781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1811.249334] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1812.025737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1812.141491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.60@tcp [ 1812.504160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1813.546211] LNet: Removed LNI 192.168.202.60@tcp [ 1813.600848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1813.690305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1813.732351] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1814.503562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.60@tcp [ 1814.726330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1814.757986] LNet: 81834:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1814.766172] LNet: 81834:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1814.775928] LNetError: 81834:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1814.784139] LNetError: 81834:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1814.815340] LNet: Removed LNI 192.168.202.60@tcp [ 1814.851394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1814.898435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1814.939345] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1815.550248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1815.619429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.60@tcp [ 1815.803128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1816.876849] LNet: Removed LNI 192.168.202.60@tcp [ 1816.942338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1817.083956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1817.113875] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1818.104126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.60@tcp [ 1819.733746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1820.779170] LNet: Removed LNI 192.168.202.60@tcp [ 1820.849885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1821.027778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1821.097945] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1822.242133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1822.344878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1822.383255] LNet: Removed LNI 192.168.202.60@tcp [ 1822.421335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1822.525245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1822.576587] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1824.058289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1832.664738] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 05:05:44 (1787389544) [ 1833.333857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1833.363989] LNet: 82973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1833.371495] LNet: 82973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1833.376090] LNetError: 82973:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1833.387699] LNetError: 82973:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1833.438523] LNet: Removed LNI 192.168.202.60@tcp [ 1833.474795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1833.554318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1833.575804] LNet: Added LNI 192.168.202.60@tcp231 [8/256/0/180] [ 1835.215614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1835.269244] LNet: Removed LNI 192.168.202.60@tcp231 [ 1835.285225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1835.366771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1835.414398] LNet: Added LNI 192.168.202.60@tcp231 [8/256/0/180] [ 1835.420767] LNet: 83119:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1836.250359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1836.336174] LNet: Removed LNI 192.168.202.60@tcp231 [ 1836.377216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1836.432149] LNet: Added LNI 192.168.202.60@tcp231 [8/256/0/180] [ 1838.129821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1838.153068] LNet: Removed LNI 192.168.202.60@tcp231 [ 1838.170673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1838.287819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1838.361442] LNet: Added LNI 192.168.202.60@tcp231 [8/256/0/0] [ 1838.373471] LNet: 83272:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1839.085524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1839.125913] LNet: Removed LNI 192.168.202.60@tcp231 [ 1839.178973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1839.254320] LNet: Added LNI 192.168.202.60@tcp231 [8/256/0/0] [ 1841.392494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1841.453487] LNet: Removed LNI 192.168.202.60@tcp231 [ 1841.489685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1841.634887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1841.707897] LNet: Added LNI 192.168.202.60@tcp231 [8/256/0/60] [ 1841.716167] LNet: 83423:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1842.436993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1842.495779] LNet: Removed LNI 192.168.202.60@tcp231 [ 1842.538469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1842.592197] LNet: Added LNI 192.168.202.60@tcp231 [8/256/0/60] [ 1844.894359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1844.970960] LNet: Removed LNI 192.168.202.60@tcp231 [ 1845.023142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1845.179549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1845.244572] LNet: Added LNI 192.168.202.60@tcp231 [8/256/0/180] [ 1845.254134] LNet: 83575:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1846.044403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1846.104049] LNet: Removed LNI 192.168.202.60@tcp231 [ 1846.144020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1846.194496] LNet: Added LNI 192.168.202.60@tcp231 [8/256/0/180] [ 1858.057302] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 05:06:09 (1787389569) [ 1858.261702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1858.319506] LNet: Removed LNI 192.168.202.60@tcp231 [ 1858.345586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1858.417172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1858.449410] LNet: Added LNI 192.168.202.60@tcp232 [8/256/0/180] [ 1859.595060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1859.645919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1859.681410] LNet: Removed LNI 192.168.202.60@tcp232 [ 1859.695668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1859.734302] LNet: Added LNI 192.168.202.60@tcp232 [8/256/0/180] [ 1860.838931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1860.926435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1860.977894] LNet: Removed LNI 192.168.202.60@tcp232 [ 1861.009396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1861.066569] LNet: Added LNI 192.168.202.60@tcp232 [8/256/0/180] [ 1862.297765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1862.384668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1862.442796] LNet: Removed LNI 192.168.202.60@tcp232 [ 1862.476507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1862.526376] LNet: Added LNI 192.168.202.60@tcp232 [8/256/0/180] [ 1863.651829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1863.686391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1863.715777] LNet: Removed LNI 192.168.202.60@tcp232 [ 1863.751670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1872.786489] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 05:06:24 (1787389584) [ 1873.332649] LNet: 85415:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1873.339116] LNet: 85415:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1874.494610] Key type .llcrypt unregistered [ 1874.497534] Key type ._llcrypt unregistered [ 1882.129736] Key type ._llcrypt registered [ 1882.132894] Key type .llcrypt registered [ 1882.167665] LNet: 85644:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1882.167665] mask usage: [+|-] ... [ 1882.382603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1882.407714] alg: No test for adler32 (adler32-zlib) [ 1883.410449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1883.453217] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1892.460697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.160@tcp [ 1904.584219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1904.634767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1904.672398] LNet: Added LNI 192.168.202.60@tcp1 [8/256/0/180] [ 1906.095605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp -m GET -e local_interrupt -r 2 [ 1906.135861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp1 -m GET -e local_interrupt -r 2 [ 1906.198120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp -m GET -e local_interrupt -r 2 [ 1906.285644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp1 -m GET -e local_interrupt -r 2 [ 1906.349686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp1 -d 192.168.202.160@tcp -m GET -e local_interrupt -r 2 [ 1906.401859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp1 -d 192.168.202.160@tcp1 -m GET -e local_interrupt -r 2 [ 1906.440854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp1 -d 192.168.202.60@tcp -m GET -e local_interrupt -r 2 [ 1906.509507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp1 -d 192.168.202.60@tcp1 -m GET -e local_interrupt -r 2 [ 1906.573382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.60@tcp 192.168.202.160@tcp [ 1906.615744] LNet: There was an unexpected network error while writing to 192.168.202.160: rc = -22 [ 1906.625077] LNet: 1 local NIs in recovery (showing 1): 192.168.202.60@tcp [ 1906.997490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1907.044275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1907.075710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1907.117936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1907.157897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1907.218409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1907.287603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1907.327716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1907.349941] LNet: 86079:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1907.383459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1907.447264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1907.497203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1907.588566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1907.633521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1907.662516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1908.476220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp -m GET -e local_dropped -r 2 [ 1908.542816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp1 -m GET -e local_dropped -r 2 [ 1908.623565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp -m GET -e local_dropped -r 2 [ 1908.679670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp1 -m GET -e local_dropped -r 2 [ 1908.749975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp1 -d 192.168.202.160@tcp -m GET -e local_dropped -r 2 [ 1908.823096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp1 -d 192.168.202.160@tcp1 -m GET -e local_dropped -r 2 [ 1908.879854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp1 -d 192.168.202.60@tcp -m GET -e local_dropped -r 2 [ 1908.944533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp1 -d 192.168.202.60@tcp1 -m GET -e local_dropped -r 2 [ 1909.017865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.60@tcp 192.168.202.160@tcp [ 1909.092559] LNet: There was an unexpected network error while writing to 192.168.202.160: rc = -22 [ 1909.606476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1909.650957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1909.721167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1909.832317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1909.911847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1909.960468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1910.027750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1910.096469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1910.133811] LNet: 86200:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1910.197725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1910.290467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1910.335811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1910.394416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1910.437785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1910.471275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1911.485956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp -m GET -e local_aborted -r 2 [ 1911.586189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp1 -m GET -e local_aborted -r 2 [ 1911.673730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp -m GET -e local_aborted -r 2 [ 1911.771732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp1 -m GET -e local_aborted -r 2 [ 1911.838890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp1 -d 192.168.202.160@tcp -m GET -e local_aborted -r 2 [ 1911.930688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp1 -d 192.168.202.160@tcp1 -m GET -e local_aborted -r 2 [ 1911.997580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp1 -d 192.168.202.60@tcp -m GET -e local_aborted -r 2 [ 1912.071221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp1 -d 192.168.202.60@tcp1 -m GET -e local_aborted -r 2 [ 1912.156977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.60@tcp 192.168.202.160@tcp [ 1912.216067] LNet: There was an unexpected network error while writing to 192.168.202.160: rc = -22 [ 1912.686651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1912.749944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1912.826386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1912.885362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1913.006391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1913.064236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1913.121771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1913.200222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1913.273986] LNet: 86320:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1913.332542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1913.402399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1913.508992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1913.565965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1913.643769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1913.744680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1914.716633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp -m GET -e local_no_route -r 2 [ 1914.776941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp1 -m GET -e local_no_route -r 2 [ 1914.897068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp -m GET -e local_no_route -r 2 [ 1914.956097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp1 -m GET -e local_no_route -r 2 [ 1914.996421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp1 -d 192.168.202.160@tcp -m GET -e local_no_route -r 2 [ 1915.040826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp1 -d 192.168.202.160@tcp1 -m GET -e local_no_route -r 2 [ 1915.084856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp1 -d 192.168.202.60@tcp -m GET -e local_no_route -r 2 [ 1915.131334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp1 -d 192.168.202.60@tcp1 -m GET -e local_no_route -r 2 [ 1915.166541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.60@tcp 192.168.202.160@tcp [ 1915.183410] LNet: There was an unexpected network error while writing to 192.168.202.160: rc = -22 [ 1915.551051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1915.639459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1915.712411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1915.765685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1915.829483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1915.876924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1915.977438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1916.048499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1916.098909] LNet: 86441:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1916.141335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1916.179523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1916.233388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1916.288826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1916.365279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1916.407683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1917.154847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp -m GET -e local_timeout -r 2 [ 1917.235744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp1 -m GET -e local_timeout -r 2 [ 1917.324834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp -m GET -e local_timeout -r 2 [ 1917.409300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp1 -m GET -e local_timeout -r 2 [ 1917.459185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp1 -d 192.168.202.160@tcp -m GET -e local_timeout -r 2 [ 1917.523940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp1 -d 192.168.202.160@tcp1 -m GET -e local_timeout -r 2 [ 1917.569196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp1 -d 192.168.202.60@tcp -m GET -e local_timeout -r 2 [ 1917.629866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp1 -d 192.168.202.60@tcp1 -m GET -e local_timeout -r 2 [ 1917.726520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.60@tcp 192.168.202.160@tcp [ 1918.254132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1918.335423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1918.421316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1918.511124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1918.632359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1918.703560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1918.744768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1918.844925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1918.926444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1919.004684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1919.066377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1919.158202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1919.247790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1919.330725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1920.197977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp -m GET -e remote_dropped -r 2 [ 1920.259930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp -d 192.168.202.160@tcp1 -m GET -e remote_dropped -r 2 [ 1920.316586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp -m GET -e remote_dropped -r 2 [ 1920.367258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp -d 192.168.202.60@tcp1 -m GET -e remote_dropped -r 2 [ 1920.436571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp1 -d 192.168.202.160@tcp -m GET -e remote_dropped -r 2 [ 1920.495403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp1 -d 192.168.202.160@tcp1 -m GET -e remote_dropped -r 2 [ 1920.539297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.60@tcp1 -d 192.168.202.60@tcp -m GET -e remote_dropped -r 2 [ 1920.587815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.60@tcp1 -d 192.168.202.60@tcp1 -m GET -e remote_dropped -r 2 [ 1920.675905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.60@tcp 192.168.202.160@tcp [ 1920.707267] LNet: There was an unexpected network error while writing to 192.168.202.160: rc = -22 [ 1920.725623] LNet: Skipped 1 previous similar message [ 1920.727834] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.160@tcp [ 1921.113652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1921.143942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1921.174717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1921.223096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1921.259200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1921.310164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1921.346267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1921.386807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1921.417253] LNet: 86681:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1921.427406] LNet: 86681:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1921.454370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1921.512415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1921.580917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1921.633628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1921.673352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1921.715657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1928.748295] LNet: 86756:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1928.754397] LNetError: 86756:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1929.823179] LNet: Removed LNI 192.168.202.60@tcp [ 1929.826193] LNetError: 86756:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1929.841816] LNet: Removed LNI 192.168.202.60@tcp1 [ 1930.360706] Key type .llcrypt unregistered [ 1930.365473] Key type ._llcrypt unregistered [ 1953.168876] Key type ._llcrypt registered [ 1953.174155] Key type .llcrypt registered [ 1955.050181] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 05:07:46 (1787389666) [ 1955.854902] LNet: 87565:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1955.854902] mask usage: [+|-] ... [ 1956.055429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1956.080111] alg: No test for adler32 (adler32-zlib) [ 1957.027718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1957.078729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1957.138110] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1958.221941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1958.248154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.60@tcp [ 1958.283280] LNetError: 87737:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.202.60@tcp to recovery [ 1958.397187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.60@tcp [ 1958.542990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1966.191590] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 05:07:57 (1787389677) [ 1966.342257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1966.369714] LNet: 88322:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1966.386648] LNetError: 88322:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1966.431174] LNet: Removed LNI 192.168.202.60@tcp [ 1966.477123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1966.569888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1966.606329] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1967.193332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1967.213599] LNet: Added LNI 192.168.202.60@tcp1 [8/256/0/180] [ 1968.182707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.60@tcp [ 1968.195727] LNetError: 88594:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.202.60@tcp to recovery [ 1968.607319] LNet: 1 local NIs in recovery (showing 1): 192.168.202.60@tcp [ 1968.612351] LNet: There was an unexpected network error while writing to 192.168.202.60: rc = -22 [ 1969.456623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.60@tcp [ 1979.067145] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 05:08:10 (1787389690) [ 1981.273908] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg260-server [ 1983.466769] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 05:08:15 (1787389695) [ 1985.964758] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg260-server [ 1988.349220] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 05:08:20 (1787389700) [ 1990.482637] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg260-server [ 1993.340617] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 05:08:24 (1787389704) [ 1993.445217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1993.475660] LNet: 90210:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1993.490507] LNetError: 90210:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1993.503202] LNet: Removed LNI 192.168.202.60@tcp [ 1993.519701] LNet: Removed LNI 192.168.202.60@tcp1 [ 1993.562662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1993.689601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1993.719542] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 1994.729084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1994.820618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1994.883568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1994.933236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1995.021822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1995.080801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 2003.719133] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 05:08:35 (1787389715) [ 2003.863353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2003.901355] LNet: 90937:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2003.905292] LNetError: 90937:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2003.918811] LNetError: 90937:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2003.940660] LNet: Removed LNI 192.168.202.60@tcp [ 2003.967802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2004.028597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2004.045303] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 2012.892229] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 05:08:44 (1787389724) [ 2013.082152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2013.149992] LNet: 91644:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2013.162078] LNetError: 91644:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2013.236447] LNet: Removed LNI 192.168.202.60@tcp [ 2013.312561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2013.408933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2013.453296] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 2023.247103] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 05:08:55 (1787389735) [ 2025.285209] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 2027.633094] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 05:08:59 (1787389739) [ 2028.359926] LNet: 92683:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2028.372764] LNetError: 92683:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2028.418949] LNet: Removed LNI 192.168.202.60@tcp [ 2029.278837] Key type .llcrypt unregistered [ 2029.281179] Key type ._llcrypt unregistered [ 2037.424592] Key type ._llcrypt registered [ 2037.426261] Key type .llcrypt registered [ 2037.521960] LNet: 92912:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2037.521960] mask usage: [+|-] ... [ 2037.807928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2037.855829] alg: No test for adler32 (adler32-zlib) [ 2038.843268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2038.886378] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 2049.336194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.160@tcp [ 2055.197742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.160@tcp [ 2055.788493] LNet: 93101:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2055.803444] LNetError: 93101:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2055.830521] LNet: Removed LNI 192.168.202.60@tcp [ 2056.234574] Key type .llcrypt unregistered [ 2056.236974] Key type ._llcrypt unregistered [ 2071.002270] Key type ._llcrypt registered [ 2071.008312] Key type .llcrypt registered [ 2073.168540] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2075.375559] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2077.205815] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 05:09:49 (1787389789) [ 2079.284729] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2081.633990] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 05:09:53 (1787389793) [ 2084.101661] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg260-server [ 2085.782468] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 05:09:58 (1787389798) [ 2088.226371] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg260-server [ 2090.213309] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 05:10:02 (1787389802) [ 2091.472630] LNet: 95152:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2091.472630] mask usage: [+|-] ... [ 2092.282881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2092.300782] alg: No test for adler32 (adler32-zlib) [ 2093.391552] Lustre: Lustre: Build Version: 2.17.54_110_g2f98931 [ 2093.635317] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 2095.328487] Key type lgssc registered [ 2097.732715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2125.186224] Key type lgssc unregistered [ 2125.562772] LNet: 96277:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2125.583810] LNetError: 96277:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2125.610868] LNet: Removed LNI 192.168.202.60@tcp [ 2126.628591] Key type .llcrypt unregistered [ 2126.631390] Key type ._llcrypt unregistered [ 2146.880471] Key type ._llcrypt registered [ 2146.883281] Key type .llcrypt registered [ 2148.296680] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 05:11:00 (1787389860) [ 2148.756881] LNet: 97075:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2148.756881] mask usage: [+|-] ... [ 2148.884228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2148.897927] alg: No test for adler32 (adler32-zlib) [ 2149.892395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2149.975454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2150.078993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2150.156765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2150.200972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2158.110610] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 05:11:09 (1787389869) [ 2158.271385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2158.315171] LNet: 97687:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2158.361696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2158.512128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2158.563599] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 2159.629955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2159.676249] LNet: 97837:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2159.685610] LNetError: 97837:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2159.706916] LNet: Removed LNI 192.168.202.60@tcp [ 2159.734772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2159.836614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2159.884773] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 2160.680589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2160.706309] LNet: 97866:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2160.723725] LNetError: 97866:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2160.738848] LNet: Removed LNI 192.168.202.60@tcp [ 2160.762541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2160.849686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2160.882748] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 2169.629781] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 05:11:21 (1787389881) [ 2169.980559] LNet: 98456:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2169.988179] LNetError: 98456:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2170.002279] LNet: Removed LNI 192.168.202.60@tcp [ 2170.988712] Key type .llcrypt unregistered [ 2170.991132] Key type ._llcrypt unregistered [ 2179.730099] Key type ._llcrypt registered [ 2179.733592] Key type .llcrypt registered [ 2179.774739] LNet: 98687:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2179.774739] mask usage: [+|-] ... [ 2179.981435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2180.000974] alg: No test for adler32 (adler32-zlib) [ 2180.975785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2181.077164] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 2181.083383] LNet: *** cfs_fail_loc=e003, val=100*** [ 2181.966314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.60@tcp [ 2181.993402] LNet: *** cfs_fail_loc=e003, val=100*** [ 2181.995843] LNet: Skipped 2 previous similar messages [ 2182.008734] Ping reply received with mapping: [ 2182.016802] found metadata reply NI[192.168.202.60@tcp] [ 2182.025154] 3c ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2182.034304] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2182.037995] 5a 5a 5a 5a 5a 5a 5a 5a [ 2182.226643] LNet: 98855:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2182.232950] LNetError: 98855:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2182.256904] LNet: Removed LNI 192.168.202.60@tcp [ 2182.954281] Key type .llcrypt unregistered [ 2182.959315] Key type ._llcrypt unregistered [ 2197.472421] Key type ._llcrypt registered [ 2197.477604] Key type .llcrypt registered [ 2199.265298] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 05:11:50 (1787389910) [ 2200.074582] Key type .llcrypt unregistered [ 2200.079715] Key type ._llcrypt unregistered [ 2210.516801] Key type ._llcrypt registered [ 2210.518390] Key type .llcrypt registered [ 2210.722834] LNet: 99851:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2210.722834] mask usage: [+|-] ... [ 2211.253035] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2211.266744] alg: No test for adler32 (adler32-zlib) [ 2212.705657] LNetError: 99901:0:(api-ni.c:2779:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2213.541217] Key type .llcrypt unregistered [ 2213.542978] Key type ._llcrypt unregistered [ 2222.434564] Key type ._llcrypt registered [ 2222.441025] Key type .llcrypt registered [ 2224.720284] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 05:12:15 (1787389935) [ 2225.965396] Key type .llcrypt unregistered [ 2225.982744] Key type ._llcrypt unregistered [ 2236.427272] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2236.445322] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2238.417896] Key type ._llcrypt registered [ 2238.428905] Key type .llcrypt registered [ 2238.520199] LNet: 100762:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2238.520199] mask usage: [+|-] ... [ 2238.785928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2238.806711] alg: No test for adler32 (adler32-zlib) [ 2239.829651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2239.943051] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2251.550162] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 05:12:43 (1787389963) [ 2251.753654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2251.802902] LNet: 101673:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2251.813232] LNetError: 101673:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2251.830524] LNet: Removed LNI 10.1.2.3@tcp [ 2251.861631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2252.894333] Key type .llcrypt unregistered [ 2252.896917] Key type ._llcrypt unregistered [ 2260.747495] Key type ._llcrypt registered [ 2260.756375] Key type .llcrypt registered [ 2260.858883] LNet: 101907:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2260.858883] mask usage: [+|-] ... [ 2261.290436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2261.338046] alg: No test for adler32 (adler32-zlib) [ 2262.355305] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 2262.415148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.60@tcp [ 2262.637810] LNet: 101965:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2262.652734] LNetError: 101965:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2262.688574] LNet: Removed LNI 192.168.202.60@tcp [ 2263.677488] Key type .llcrypt unregistered [ 2263.679094] Key type ._llcrypt unregistered [ 2279.350976] Key type ._llcrypt registered [ 2279.357347] Key type .llcrypt registered [ 2281.158432] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 05:13:12 (1787389992) [ 2281.767427] Key type .llcrypt unregistered [ 2281.771602] Key type ._llcrypt unregistered [ 2291.780293] Key type ._llcrypt registered [ 2291.782711] Key type .llcrypt registered [ 2291.817596] LNet: 102950:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2291.817596] mask usage: [+|-] ... [ 2291.991640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2292.009353] alg: No test for adler32 (adler32-zlib) [ 2303.196562] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 05:13:34 (1787390014) [ 2303.425540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2303.501173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2303.582733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2303.611851] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 2304.633455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2304.704664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2304.771282] LNet: Removed LNI 192.168.202.60@tcp [ 2305.017435] LNet: 103710:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2305.566483] Key type .llcrypt unregistered [ 2305.571724] Key type ._llcrypt unregistered [ 2319.825226] Key type ._llcrypt registered [ 2319.827177] Key type .llcrypt registered [ 2321.741786] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 05:13:53 (1787390033) [ 2322.068931] LNet: 104475:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2322.068931] mask usage: [+|-] ... [ 2322.276556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2322.288959] alg: No test for adler32 (adler32-zlib) [ 2323.231825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2323.299628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2323.316662] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 2324.344658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.60@tcp [ 2324.414647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.60@tcp [ 2324.428146] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2324.431713] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2324.442211] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2324, 0, 0 [ 2324.458615] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2324.467771] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2324.477447] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2324.483570] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2324, 0, 0 [ 2324.490157] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2324.491497] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2324.497557] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2324.503736] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2324, 0, 2374 [ 2324.509774] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2324.514718] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2324.525310] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2324.529361] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2324, 0, 2374 [ 2324.537325] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2332.596208] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 05:14:04 (1787390044) [ 2333.187159] LNet: 105252:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2333.193510] LNetError: 105252:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2334.249601] LNet: Removed LNI 192.168.202.60@tcp [ 2335.119071] Key type .llcrypt unregistered [ 2335.120278] Key type ._llcrypt unregistered [ 2342.120484] Key type ._llcrypt registered [ 2342.126172] Key type .llcrypt registered [ 2342.193219] LNet: 105476:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2342.193219] mask usage: [+|-] ... [ 2342.437672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2342.463403] alg: No test for adler32 (adler32-zlib) [ 2343.540814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2343.580487] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 2353.696420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.160@tcp [ 2362.396051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2362.447969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2362.483801] LNet: Added LNI 192.168.202.60@tcp1 [8/256/0/180] [ 2363.652704] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2363.753388] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2363.765049] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2363.784230] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2366.324685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2366.360737] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2379.928619] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 05:14:51 (1787390091) [ 2380.294311] LNet: 106759:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2380.305907] LNetError: 106759:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2380.317963] LNet: Removed LNI 192.168.202.60@tcp [ 2381.343310] LNet: Removed LNI 192.168.202.60@tcp1 [ 2381.344929] LNetError: 106759:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2381.353743] LNetError: 106759:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2381.370357] LNet: Removed LNI 10.1.2.3@tcp99 [ 2382.465873] Key type .llcrypt unregistered [ 2382.467790] Key type ._llcrypt unregistered [ 2391.207046] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2391.215674] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2394.004549] Key type ._llcrypt registered [ 2394.007228] Key type .llcrypt registered [ 2394.076275] LNet: 107026:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2394.076275] mask usage: [+|-] ... [ 2394.412100] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2394.453633] alg: No test for adler32 (adler32-zlib) [ 2395.456912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2395.534464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2395.575899] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 2396.701798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2396.728766] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2399.888084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2399.982276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2400.035870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2400.072241] LNetError: 107504:0:(peer.c:2234:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2400.102595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2400.174789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.60@tcp [ 2401.044487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2401.155119] LNet: 107525:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2401.158439] LNetError: 107525:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2401.163570] LNet: Removed LNI 10.1.2.3@tcp [ 2402.227339] LNet: Removed LNI 192.168.202.60@tcp [ 2403.046519] Key type .llcrypt unregistered [ 2403.047970] Key type ._llcrypt unregistered [ 2417.695293] Key type ._llcrypt registered [ 2417.701665] Key type .llcrypt registered [ 2419.567962] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 05:15:31 (1787390131) [ 2419.994147] LNet: 108324:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2419.994147] mask usage: [+|-] ... [ 2420.154139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2420.177727] alg: No test for adler32 (adler32-zlib) [ 2421.228243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2421.328620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2421.373681] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 2422.353284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.60@tcp [ 2422.417868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg260-client@tcp [ 2429.482420] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 05:15:41 (1787390141) [ 2429.645459] LNet: 109057:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2429.657407] LNetError: 109057:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2429.695695] LNet: Removed LNI 192.168.202.60@tcp [ 2430.500327] Key type .llcrypt unregistered [ 2430.501769] Key type ._llcrypt unregistered [ 2438.121819] Key type ._llcrypt registered [ 2438.127654] Key type .llcrypt registered [ 2438.292465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2438.311538] alg: No test for adler32 (adler32-zlib) [ 2439.304469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2439.339500] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 2439.395225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2446.129630] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 05:15:58 (1787390158) [ 2447.685851] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2449.502584] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 05:16:01 (1787390161) [ 2451.162325] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2452.905560] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 05:16:05 (1787390165) [ 2453.031527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2453.055168] LNet: 110473:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2453.066392] LNetError: 110473:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2453.105479] LNet: Removed LNI 192.168.202.60@tcp [ 2453.130853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2453.260166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2453.300190] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 2454.526160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2454.594187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2462.096911] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 05:16:14 (1787390174) [ 2462.300125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2462.315890] LNet: 111189:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2462.321516] LNetError: 111189:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2462.345211] LNet: Removed LNI 192.168.202.60@tcp [ 2462.359472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2462.405833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2462.430954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2463.134771] Key type .llcrypt unregistered [ 2463.139395] Key type ._llcrypt unregistered [ 2474.974789] Key type ._llcrypt registered [ 2474.978592] Key type .llcrypt registered [ 2476.333858] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 05:16:28 (1787390188) [ 2476.538990] LNet: 111993:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2476.538990] mask usage: [+|-] ... [ 2476.700699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2476.716065] alg: No test for adler32 (adler32-zlib) [ 2477.677643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2477.735289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2477.756849] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 2478.585205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2478.632258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.60@tcp [ 2485.778840] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 05:16:37 (1787390197) [ 2485.924379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2485.978049] LNet: 112742:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2486.001639] LNetError: 112742:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2486.066154] LNet: Removed LNI 192.168.202.60@tcp [ 2486.100188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2486.168554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2486.215513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2493.351973] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 05:16:45 (1787390205) [ 2493.766213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2493.823059] LNet: 113325:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2493.848554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2493.964597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2494.237133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2494.259809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2494.297842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2494.620073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2494.662207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2494.741892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2494.807313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2494.914975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2495.446247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2495.499835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2495.589570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2496.168327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2496.234225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2502.883520] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 05:16:55 (1787390215) [ 2503.411994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2503.435960] LNet: 113958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2503.452110] LNet: 113958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2503.481691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2503.544982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2503.585859] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 2504.495116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2504.560824] LNet: Added LNI 192.168.202.60@tcp2 [8/256/0/180] [ 2505.984318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2506.130842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.60@tcp [ 2506.499699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2506.533447] LNet: 114238:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2506.545513] LNetError: 114238:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2506.558060] LNet: Removed LNI 192.168.202.60@tcp [ 2506.582455] LNet: Removed LNI 192.168.202.60@tcp2 [ 2506.622419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2506.739882] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 2506.768110] LNet: Added LNI 192.168.202.60@tcp2 [8/256/0/180] [ 2506.784536] LNet: 114244:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2507.918210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.60@tcp [ 2508.270106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2508.319435] LNetError: 114281:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2508.333273] LNetError: 114281:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2508.351657] LNet: Removed LNI 192.168.202.60@tcp [ 2508.377460] LNet: Removed LNI 192.168.202.60@tcp2 [ 2508.426432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2508.549689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2508.610480] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 2509.445690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2509.463450] LNet: Added LNI 192.168.202.60@tcp2 [8/256/0/180] [ 2510.617564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.60@tcp [ 2510.722237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2510.830929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.60@tcp [ 2511.102961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2511.125406] LNet: 114566:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2511.130545] LNet: 114566:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2511.135406] LNetError: 114566:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2511.138451] LNetError: 114566:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2511.146660] LNet: Removed LNI 192.168.202.60@tcp [ 2512.166978] LNet: Removed LNI 192.168.202.60@tcp2 [ 2512.215378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2512.329990] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 2512.338133] LNet: Added LNI 192.168.202.60@tcp2 [8/256/0/180] [ 2512.346978] LNet: 114572:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2513.016882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.60@tcp [ 2513.316306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2513.333598] LNet: Removed LNI 192.168.202.60@tcp [ 2514.405325] LNet: Removed LNI 192.168.202.60@tcp2 [ 2514.457479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2514.595969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2514.625504] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 2515.351728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2515.369617] LNet: Added LNI 192.168.202.60@tcp2 [8/256/0/180] [ 2516.136666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2516.218388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.60@tcp [ 2516.489301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2516.510630] LNetError: 114888:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2516.518569] LNetError: 114888:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2516.526906] LNet: Removed LNI 192.168.202.60@tcp [ 2516.540319] LNet: Removed LNI 192.168.202.60@tcp2 [ 2516.596800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2516.713911] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 2516.719845] LNet: Added LNI 192.168.202.60@tcp2 [8/256/0/180] [ 2516.724718] LNet: 114894:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2517.230957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.60@tcp [ 2517.475512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2518.559187] LNet: Removed LNI 192.168.202.60@tcp [ 2518.584949] LNet: Removed LNI 192.168.202.60@tcp2 [ 2518.621668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2518.692418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2518.729717] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 2519.465394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2519.503487] LNet: Added LNI 192.168.202.60@tcp2 [8/256/0/180] [ 2520.741903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.60@tcp [ 2520.796627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2520.855902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.60@tcp [ 2521.194737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2521.224042] LNet: 115216:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2521.230174] LNet: 115216:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2522.271527] LNet: Removed LNI 192.168.202.60@tcp [ 2523.366868] LNet: Removed LNI 192.168.202.60@tcp2 [ 2523.418782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2523.545945] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 2523.550636] LNet: Added LNI 192.168.202.60@tcp2 [8/256/0/180] [ 2523.560439] LNet: 115222:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2524.371948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.60@tcp [ 2524.662556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2524.689782] LNetError: 115260:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2524.701888] LNetError: 115260:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2524.710912] LNet: Removed LNI 192.168.202.60@tcp [ 2525.733688] LNet: Removed LNI 192.168.202.60@tcp2 [ 2525.818374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2525.965218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2526.031656] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 2526.994795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2527.048790] LNet: Added LNI 192.168.202.60@tcp2 [8/256/0/180] [ 2528.170425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2528.225062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2528.303154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.60@tcp [ 2528.816971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2529.887489] LNet: Removed LNI 192.168.202.60@tcp [ 2529.906342] LNet: Removed LNI 192.168.202.60@tcp2 [ 2529.952561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2530.040557] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 2530.046310] LNet: Added LNI 192.168.202.60@tcp2 [8/256/0/180] [ 2530.050957] LNet: 115562:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2530.750221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.60@tcp [ 2531.177509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2532.255250] LNet: Removed LNI 192.168.202.60@tcp [ 2532.274122] LNet: Removed LNI 192.168.202.60@tcp2 [ 2532.311909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2532.406450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2532.435767] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 2533.185784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2533.204326] LNet: Added LNI 192.168.202.60@tcp2 [8/256/0/180] [ 2534.312274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.60@tcp [ 2534.407753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2534.449662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2534.540912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.60@tcp [ 2534.925388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2535.967273] LNet: Removed LNI 192.168.202.60@tcp [ 2537.012583] LNet: Removed LNI 192.168.202.60@tcp2 [ 2537.064229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2537.135866] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 2537.143807] LNet: Added LNI 192.168.202.60@tcp2 [8/256/0/180] [ 2537.983807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.60@tcp [ 2546.486938] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2548.629847] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 05:17:40 (1787390260) [ 2548.918424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2548.944862] LNet: 116681:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2548.950484] LNet: 116681:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2548.957713] LNetError: 116681:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2548.962551] LNetError: 116681:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2548.972560] LNet: Removed LNI 192.168.202.60@tcp [ 2550.001647] LNet: Removed LNI 192.168.202.60@tcp2 [ 2550.104217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2550.461260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2550.500061] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 2551.587994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2551.608706] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2552.419924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2552.437833] LNet: Removed LNI 10.1.2.3@tcp [ 2552.446991] LNet: Removed LNI 192.168.202.60@tcp [ 2552.463312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2552.522369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2552.581283] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 2552.589671] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2553.407593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2553.442421] LNet: Removed LNI 192.168.202.60@tcp [ 2553.457885] LNet: Removed LNI 10.1.2.3@tcp [ 2560.886365] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 05:17:52 (1787390272) [ 2561.419142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2561.493243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2561.600306] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2561.827706] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2561.836916] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2561.852671] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2564.326359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2564.383073] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 2565.211528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2565.245097] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2567.982829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.60@tcp [ 2568.050379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2568.134224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.60@tcp -m PUT -e local_timeout -r 1 [ 2568.253141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2568.340859] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2568.352330] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2568.672529] LNetError: 117618:0:(lib-move.c:4714:lnet_parse()) 192.168.202.60@tcp, src 192.168.202.60@tcp: Dropping PUT (error -108 looking up sender) [ 2569.697247] LNet: Removed LNI 10.1.2.3@tcp [ 2570.735414] LNet: Removed LNI 192.168.202.60@tcp [ 2571.288580] Key type .llcrypt unregistered [ 2571.295193] Key type ._llcrypt unregistered [ 2584.572771] Key type ._llcrypt registered [ 2584.578224] Key type .llcrypt registered [ 2585.841650] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 05:18:18 (1787390298) [ 2586.410808] LNet: 118847:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2586.410808] mask usage: [+|-] ... [ 2586.575366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2586.592839] alg: No test for adler32 (adler32-zlib) [ 2587.619667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2588.235814] LNet: 118928:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2589.068533] Key type .llcrypt unregistered [ 2589.073788] Key type ._llcrypt unregistered [ 2595.961197] Key type ._llcrypt registered [ 2595.965980] Key type .llcrypt registered [ 2596.017325] LNet: 119157:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2596.017325] mask usage: [+|-] ... [ 2596.259410] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2596.287249] alg: No test for adler32 (adler32-zlib) [ 2597.298990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2597.348609] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 2605.244450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.160@tcp [ 2605.450204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2606.001391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.160@tcp [ 2606.282712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2606.314479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2606.360526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2606.406613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2606.453353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2606.503031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2606.553649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2606.559621] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.160@tcp [ 2606.590866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2606.618635] LNet: 119378:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2606.648221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2606.703059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2606.741927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2606.775399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2606.808155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2606.836842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2607.380312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2607.634214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2607.660987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2607.687822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2607.729795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2607.768969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2607.812546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2607.845880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2607.883988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2607.908359] LNet: 119495:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2607.936986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2607.974133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2608.020941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2608.058768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2608.093641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2608.154862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2608.990364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.160@tcp [ 2609.210834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2609.253511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2609.278913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2609.306217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2609.330166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2609.356954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2609.387996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2609.415327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2609.440319] LNet: 119612:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2609.465803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2609.497033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2609.532826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2609.573797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2609.610463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2609.649865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2610.382271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2610.629836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2610.654175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2610.689145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2610.723344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2610.772430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2610.827824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2610.875332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2610.910496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2610.952431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2610.978354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2611.003789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2611.052887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2611.086339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2611.119340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2611.526890] LNet: 119775:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2611.532194] LNetError: 119775:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2611.546992] LNet: Removed LNI 192.168.202.60@tcp [ 2611.956677] Key type .llcrypt unregistered [ 2611.959787] Key type ._llcrypt unregistered [ 2628.303605] Key type ._llcrypt registered [ 2628.306820] Key type .llcrypt registered [ 2630.001787] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 05:19:01 (1787390341) [ 2630.241177] LNet: 120579:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2630.241177] mask usage: [+|-] ... [ 2630.473979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2630.517922] alg: No test for adler32 (adler32-zlib) [ 2631.495644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2632.023978] LNet: 120663:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2632.838378] Key type .llcrypt unregistered [ 2632.842072] Key type ._llcrypt unregistered [ 2639.817950] Key type ._llcrypt registered [ 2639.821126] Key type .llcrypt registered [ 2639.865850] LNet: 120892:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2639.865850] mask usage: [+|-] ... [ 2640.043604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2640.060295] alg: No test for adler32 (adler32-zlib) [ 2641.014084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2641.045542] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 2648.809703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.160@tcp [ 2658.953621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2659.025697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2659.056693] LNet: Added LNI 192.168.202.60@tcp1 [8/256/0/180] [ 2660.226769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2660.794965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.160@tcp [ 2661.091520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2661.119260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2661.154633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2661.188104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2661.216144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2661.249186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2661.300698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2661.350471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2661.372973] LNet: 121315:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2661.393115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2661.427412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2661.459410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2661.507274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2661.536721] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.160@tcp [ 2661.554847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2661.586220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2662.451637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2662.724315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2662.763885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2662.801086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2662.850742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2662.892205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2662.954915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2663.041705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2663.098682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2663.118341] LNet: 121433:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2663.140884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2663.207285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2663.238821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2663.295195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2663.378402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2663.489417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2664.510879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.160@tcp [ 2664.906449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2664.972460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2665.019154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2665.067290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2665.134489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2665.199578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2665.257043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2665.310693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2665.350849] LNet: 121550:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2665.379381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2665.431480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2665.468124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2665.506582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2665.539582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2665.569442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2666.409780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2666.729638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2666.780036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2666.814506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2666.878819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2666.910444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2666.958256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2667.011064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2667.051470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2667.082624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2667.110449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2667.133480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2667.175136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2667.230545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2667.293560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2672.333578] LNet: 121748:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2672.355403] LNetError: 121748:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2672.367934] LNet: Removed LNI 192.168.202.60@tcp [ 2672.385879] LNet: Removed LNI 192.168.202.60@tcp1 [ 2672.900982] Key type .llcrypt unregistered [ 2672.905323] Key type ._llcrypt unregistered [ 2692.665417] Key type ._llcrypt registered [ 2692.667762] Key type .llcrypt registered [ 2694.048405] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 05:20:06 (1787390406) [ 2702.386803] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 05:20:14 (1787390414) [ 2710.816543] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 05:20:22 (1787390422) [ 2719.448660] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 05:20:31 (1787390431) [ 2726.665627] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 05:20:39 (1787390439) [ 2734.924540] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 05:20:47 (1787390447) [ 2742.475917] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 05:20:54 (1787390454) [ 2751.336359] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 05:21:03 (1787390463) [ 2758.715561] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 05:21:10 (1787390470) [ 2766.933251] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 05:21:19 (1787390479) [ 2774.842544] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 05:21:27 (1787390487) [ 2782.796413] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 05:21:34 (1787390494) [ 2790.439866] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 05:21:42 (1787390502) [ 2799.063568] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 05:21:50 (1787390510) [ 2806.967346] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 05:21:59 (1787390519) [ 2814.627641] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 05:22:06 (1787390526) [ 2822.612312] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 05:22:14 (1787390534) [ 2830.626960] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 05:22:22 (1787390542) [ 2830.836919] LNet: 132290:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2830.836919] mask usage: [+|-] ... [ 2830.970081] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2830.987719] alg: No test for adler32 (adler32-zlib) [ 2831.916395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2832.042420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2832.062491] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 2833.456279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2833.494522] LNet: 132463:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2833.504084] LNetError: 132463:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2833.535854] LNet: Removed LNI 192.168.202.60@tcp [ 2833.592980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2833.647784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2833.667333] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 2840.643508] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 05:22:32 (1787390552) [ 2847.254368] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 05:22:39 (1787390559) [ 2855.465411] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 05:22:47 (1787390567) [ 2862.671078] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 05:22:54 (1787390574) [ 2870.078314] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 05:23:02 (1787390582) [ 2878.142924] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 05:23:10 (1787390590) [ 2886.473305] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 05:23:18 (1787390598) [ 2893.983157] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 05:23:26 (1787390606) [ 2901.505546] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 05:23:33 (1787390613) [ 2909.429784] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 05:23:41 (1787390621) [ 2916.322651] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 05:23:48 (1787390628) [ 2924.158181] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 05:23:56 (1787390636) [ 2933.173686] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2654 sec ========== 05:24:04 (1787390644) [ 2933.927478] LNet: 139927:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2933.935435] LNetError: 139927:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2933.953202] LNet: Removed LNI 192.168.202.60@tcp [ 2934.835512] Key type .llcrypt unregistered [ 2934.841026] Key type ._llcrypt unregistered [ 2950.245238] Key type ._llcrypt registered [ 2950.253548] Key type .llcrypt registered [ 2950.417633] LNet: 140404:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2950.417633] mask usage: [+|-] ... [ 2950.646524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2950.665780] alg: No test for adler32 (adler32-zlib) [ 2951.711706] Lustre: Lustre: Build Version: 2.17.54_110_g2f98931 [ 2952.117799] LNet: Added LNI 192.168.202.60@tcp [8/256/0/180] [ 2953.928149] Key type lgssc registered [ 2956.328698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3033.782940] Lustre: Mounted lustre-client [ 3033.987334] LNet: 141876:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 3033.987334] mask usage: [+|-] ... [ 3039.981504] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3048.741599] Lustre: Unmounted lustre-client [ 3074.127621] Key type lgssc unregistered [ 3074.527438] LNet: 142857:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3074.540695] LNetError: 142857:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3074.559803] LNet: Removed LNI 192.168.202.60@tcp [ 3075.682614] Key type .llcrypt unregistered [ 3075.693091] Key type ._llcrypt unregistered