[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 453276244 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002263] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004008] kvm-guest: setup PV IPIs [ 0.005990] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007008] pid_max: default: 32768 minimum: 301 [ 0.008104] LSM: Security Framework initializing [ 0.009036] Yama: becoming mindful. [ 0.010022] SELinux: Initializing. [ 0.011057] *** VALIDATE selinux *** [ 0.018059] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023363] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024162] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026127] *** VALIDATE tmpfs *** [ 0.027495] *** VALIDATE proc *** [ 0.029127] *** VALIDATE cgroup *** [ 0.030014] *** VALIDATE cgroup2 *** [ 0.031297] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032201] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033014] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034033] Spectre V2 : User space: Vulnerable [ 0.035008] Speculative Store Bypass: Vulnerable [ 0.038440] debug: unmapping init [mem 0xffffffff85c59000-0xffffffff85c60fff] [ 0.041191] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042734] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043034] ... version: 2 [ 0.044016] ... bit width: 48 [ 0.045013] ... generic registers: 4 [ 0.046013] ... value mask: 0000ffffffffffff [ 0.047016] ... max period: 00007fffffffffff [ 0.048016] ... fixed-purpose events: 3 [ 0.049013] ... event mask: 000000070000000f [ 0.050321] rcu: Hierarchical SRCU implementation. [ 0.052493] smp: Bringing up secondary CPUs ... [ 0.053616] x86: Booting SMP configuration: [ 0.054028] .... node #0, CPUs: #1 #2 #3 [ 0.061226] smp: Brought up 1 node, 4 CPUs [ 0.063017] smpboot: Max logical packages: 1 [ 0.064012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.131549] node 0 deferred pages initialised in 65ms [ 0.137252] devtmpfs: initialized [ 0.138325] x86/mm: Memory block size: 128MB [ 0.142618] gcov: version magic: 0x41383552 [ 0.145106] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.149158] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.153511] pinctrl core: initialized pinctrl subsystem [ 0.157384] [ 0.160020] ************************************************************* [ 0.162018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164040] ** ** [ 0.166027] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.169022] ** ** [ 0.171019] ** This means that this kernel is built to expose internal ** [ 0.173022] ** IOMMU data structures, which may compromise security on ** [ 0.176033] ** your system. ** [ 0.178019] ** ** [ 0.179019] ** If you see this message and you are not debugging the ** [ 0.181014] ** kernel, report this immediately to your vendor! ** [ 0.183016] ** ** [ 0.184015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.186014] ************************************************************* [ 0.188962] NET: Registered protocol family 16 [ 0.191506] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.194068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.197083] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.201143] cpuidle: using governor menu [ 0.202902] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.205514] PCI: Using configuration type 1 for base access [ 0.207136] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.217185] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.221026] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.225039] cryptd: max_cpu_qlen set to 1000 [ 0.227228] ACPI: Added _OSI(Module Device) [ 0.228013] ACPI: Added _OSI(Processor Device) [ 0.230020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.231022] ACPI: Added _OSI(Processor Aggregator Device) [ 0.236197] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.241531] ACPI: Interpreter enabled [ 0.243068] ACPI: PM: (supports S0 S3 S4 S5) [ 0.245019] ACPI: Using IOAPIC for interrupt routing [ 0.246153] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.249390] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.259906] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.262057] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.265024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.268119] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.272732] acpiphp: Slot [2] registered [ 0.274217] acpiphp: Slot [5] registered [ 0.276182] acpiphp: Slot [6] registered [ 0.277176] acpiphp: Slot [3] registered [ 0.279129] acpiphp: Slot [4] registered [ 0.280121] acpiphp: Slot [7] registered [ 0.281112] acpiphp: Slot [8] registered [ 0.283121] acpiphp: Slot [9] registered [ 0.284118] acpiphp: Slot [10] registered [ 0.286152] acpiphp: Slot [11] registered [ 0.287121] acpiphp: Slot [12] registered [ 0.288121] acpiphp: Slot [13] registered [ 0.290127] acpiphp: Slot [14] registered [ 0.291111] acpiphp: Slot [15] registered [ 0.293134] acpiphp: Slot [16] registered [ 0.294150] acpiphp: Slot [17] registered [ 0.296173] acpiphp: Slot [18] registered [ 0.297138] acpiphp: Slot [19] registered [ 0.299125] acpiphp: Slot [20] registered [ 0.300164] acpiphp: Slot [21] registered [ 0.301133] acpiphp: Slot [22] registered [ 0.303127] acpiphp: Slot [23] registered [ 0.304163] acpiphp: Slot [24] registered [ 0.306132] acpiphp: Slot [25] registered [ 0.307117] acpiphp: Slot [26] registered [ 0.309126] acpiphp: Slot [27] registered [ 0.310118] acpiphp: Slot [28] registered [ 0.311118] acpiphp: Slot [29] registered [ 0.313189] acpiphp: Slot [30] registered [ 0.314134] acpiphp: Slot [31] registered [ 0.316080] PCI host bridge to bus 0000:00 [ 0.317030] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.319050] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.321032] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.324026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.326023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.329046] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.331256] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.334091] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.337395] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.344487] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.348034] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.351024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.353022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.355030] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.356481] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.359729] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.362049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.364854] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.369013] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.378958] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.383017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.386000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.393016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.397015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.405022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.414175] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.423025] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.436024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.453024] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.476046] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.478448] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.481494] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.483397] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.486220] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.491000] iommu: Default domain type: Passthrough [ 0.494549] SCSI subsystem initialized [ 0.497231] ACPI: bus type USB registered [ 0.498141] usbcore: registered new interface driver usbfs [ 0.501128] usbcore: registered new interface driver hub [ 0.503105] usbcore: registered new device driver usb [ 0.505271] pps_core: LinuxPPS API ver. 1 registered [ 0.508016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.511072] PTP clock support registered [ 0.514056] EDAC MC: Ver: 3.0.0 [ 0.516234] PCI: Using ACPI for IRQ routing [ 0.518136] NetLabel: Initializing [ 0.519015] NetLabel: domain hash size = 128 [ 0.520016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.521111] NetLabel: unlabeled traffic allowed by default [ 0.522192] vgaarb: loaded [ 0.524269] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.525022] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.530436] clocksource: Switched to clocksource kvm-clock [ 0.636346] VFS: Disk quotas dquot_6.6.0 [ 0.638045] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.640349] *** VALIDATE ramfs *** [ 0.641574] *** VALIDATE hugetlbfs *** [ 0.643040] pnp: PnP ACPI init [ 0.645385] pnp: PnP ACPI: found 6 devices [ 0.668529] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.671809] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.674053] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.676123] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.678250] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.680882] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.683414] NET: Registered protocol family 2 [ 0.685345] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.689569] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.693117] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.698741] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.701946] TCP: Hash tables configured (established 65536 bind 65536) [ 0.705105] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.707577] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.710347] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.713291] NET: Registered protocol family 1 [ 0.715722] RPC: Registered named UNIX socket transport module. [ 0.717887] RPC: Registered udp transport module. [ 0.719729] RPC: Registered tcp transport module. [ 0.721630] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.724047] NET: Registered protocol family 44 [ 0.725782] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.727661] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.729060] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.730352] PCI: CLS 0 bytes, default 64 [ 0.731341] Unpacking initramfs... [ 3.567905] debug: unmapping init [mem 0xffff8a73fcc64000-0xffff8a73fffcffff] [ 3.585668] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.593310] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.600340] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.870403] Initialise system trusted keyrings [ 5.873295] Key type blacklist registered [ 5.877092] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.900593] zbud: loaded [ 5.921118] *** VALIDATE nfs *** [ 5.927197] *** VALIDATE nfs4 *** [ 5.935636] pstore: using deflate compression [ 5.962340] Platform Keyring initialized [ 6.392677] NET: Registered protocol family 38 [ 6.397818] Key type asymmetric registered [ 6.405123] Asymmetric key parser 'x509' registered [ 6.408112] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.412919] io scheduler mq-deadline registered [ 6.415926] io scheduler kyber registered [ 6.421267] io scheduler bfq registered [ 6.427559] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 6.433539] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.444803] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.449573] ACPI: Power Button [PWRF] [ 6.459097] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.468870] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.489119] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.520069] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.553589] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.571850] Non-volatile memory driver v1.3 [ 6.576712] Linux agpgart interface v0.103 [ 6.630769] virtio_blk virtio1: [vda] 145168 512-byte logical blocks (74.3 MB/70.9 MiB) [ 6.640958] vda: detected capacity change from 0 to 74326016 [ 6.692985] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.702209] vdb: detected capacity change from 0 to 1073741824 [ 6.716347] libphy: Fixed MDIO Bus: probed [ 6.762869] usbcore: registered new interface driver usbserial_generic [ 6.765997] usbserial: USB Serial support registered for generic [ 6.768695] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.774361] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.776390] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.788250] mousedev: PS/2 mouse device common for all mice [ 6.800355] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.800775] rtc_cmos 00:05: RTC can wake from S4 [ 6.824583] rtc_cmos 00:05: registered as rtc0 [ 6.831157] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.851889] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.864139] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.868301] intel_pstate: CPU model not supported [ 6.903150] hid: raw HID events driver (C) Jiri Kosina [ 6.904877] usbcore: registered new interface driver usbhid [ 6.911266] usbhid: USB HID core driver [ 6.915937] drop_monitor: Initializing network drop monitor service [ 6.923707] Initializing XFRM netlink socket [ 6.934810] NET: Registered protocol family 10 [ 6.950078] Segment Routing with IPv6 [ 6.952956] NET: Registered protocol family 17 [ 6.956646] mpls_gso: MPLS GSO support [ 6.968798] RAS: Correctable Errors collector initialized. [ 6.973214] AVX version of gcm_enc/dec engaged. [ 6.977220] AES CTR mode by8 optimization enabled [ 7.282660] sched_clock: Marking stable (7282303654, 0)->(8190820557, -908516903) [ 7.288690] registered taskstats version 1 [ 7.291814] Loading compiled-in X.509 certificates [ 7.295541] zswap: loaded using pool lzo/zbud [ 7.337628] Key type big_key registered [ 7.363813] Key type encrypted registered [ 7.365692] ima: No TPM chip found, activating TPM-bypass! [ 7.368511] ima: Allocated hash algorithm: sha1 [ 7.370539] ima: No architecture policies found [ 7.372826] evm: Initialising EVM extended attributes: [ 7.375880] evm: security.selinux [ 7.378173] evm: security.ima [ 7.379683] evm: security.capability [ 7.382652] evm: HMAC attrs: 0x1 [ 7.387533] rtc_cmos 00:05: setting system clock to 2026-06-23 12:11:30 UTC (1782216690) [ 7.397512] debug: unmapping init [mem 0xffffffff86c03000-0xffffffff86dfffff] [ 7.405246] debug: unmapping init [mem 0xffffffff85982000-0xffffffff85c58fff] [ 7.415201] Write protecting the kernel read-only data: 28672k [ 7.422865] debug: unmapping init [mem 0xffffffff84003000-0xffffffff841fffff] [ 7.430958] debug: unmapping init [mem 0xffffffff84914000-0xffffffff849fffff] [ 7.566776] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.579717] systemd[1]: Detected virtualization kvm. [ 7.586947] systemd[1]: Detected architecture x86-64. [ 7.590361] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.675721] systemd[1]: No hostname configured. [ 7.680705] systemd[1]: Set hostname to . [ 7.692542] random: systemd: uninitialized urandom read (16 bytes read) [ 7.694840] systemd[1]: Initializing machine ID from random generator. [ 7.840859] random: ln: uninitialized urandom read (6 bytes read) [ 8.036262] random: systemd: uninitialized urandom read (16 bytes read) [ 8.038355] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 8.044687] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 8.048245] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Paths. Starting Journal Service... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 9.904157] device-mapper: uevent: version 1.0.3 [ 9.908864] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 12.218076] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ 12.535343] scsi host0: ata_piix [ 12.659523] scsi host1: ata_piix [ 12.664410] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 12.669371] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 17.896747] random: crng init done [ 17.899594] random: 7 urandom warning(s) missed due to ratelimiting [ 19.733767] dracut-initqueue[586]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 21.244640] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 24.288754] printk: systemd: 26 output lines suppressed due to ratelimiting [ 25.454476] SELinux: Disabled at runtime. [ 25.585136] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 25.602743] systemd[1]: Detected virtualization kvm. [ 25.606650] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 27.488840] systemd[1]: initrd-switch-root.service: Succeeded. [ 27.499526] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 27.548182] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 27.563729] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 27.569557] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 27.612444] systemd[1]: Starting Journal Service... Starting Journal Service... [ 27.632104] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Switch Root. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ 27.778082] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. [ OK ] Reached target Paths. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... Starting udev Coldplug all Devices... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 29.381388] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 31.147983] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 31.350104] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 31.802174] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 31.982889] EDAC sbridge: Ver: 1.1.2 [ 35.555618] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit)[ 36.471953] NFS: Registering the id_resolver key type [ 36.482058] Key type id_resolver registered [ 36.483660] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg240-client login: [ 100.082088] hrtimer: interrupt took 11000048 ns [ 110.815533] libcfs: loading out-of-tree module taints kernel. [ 111.053438] Key type ._llcrypt registered [ 111.073979] Key type .llcrypt registered [ 112.486349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 112.505822] alg: No test for adler32 (adler32-zlib) [ 114.379713] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 115.830639] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 117.631204] Key type lgssc registered [ 119.723676] Lustre: Echo OBD driver; http://www.lustre.org/ [ 245.625395] Lustre: Mounted lustre-client [ 250.522348] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 266.452574] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing check_logdir /tmp/testlogs/ [ 271.327299] Lustre: lustre-OST0000-osc-ffff8a7442e64800: disconnect after 23s idle [ 272.558608] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing yml_node [ 278.468497] Lustre: DEBUG MARKER: Client: 2.17.54.83 [ 281.485520] Lustre: DEBUG MARKER: MDS: 2.17.54.83 [ 284.385686] Lustre: DEBUG MARKER: OSS: 2.17.54.83 [ 285.800669] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Jun 23 08:16:07 EDT 2026 [ 302.659632] Lustre: Unmounted lustre-client [ 320.431631] Key type lgssc unregistered [ 320.699737] LNet: 8405:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 320.712188] LNetError: 8405:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 321.768529] LNet: Removed LNI 192.168.202.40@tcp [ 322.563100] Key type .llcrypt unregistered [ 322.565519] Key type ._llcrypt unregistered [ 351.057286] Key type ._llcrypt registered [ 351.065358] Key type .llcrypt registered [ 351.123826] LNet: 8987:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 351.123826] mask usage: [+|-] ... [ 351.377112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 351.412596] alg: No test for adler32 (adler32-zlib) [ 352.382364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 352.444433] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 352.462259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 356.610571] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 356.771587] LNet: 9288:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 356.778791] LNetError: 9288:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 356.811762] LNet: Removed LNI 192.168.202.40@tcp [ 357.329801] Key type .llcrypt unregistered [ 357.332795] Key type ._llcrypt unregistered [ 364.417246] Key type ._llcrypt registered [ 364.419186] Key type .llcrypt registered [ 365.640712] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 08:17:27 (1782217047) [ 365.851887] LNet: 9681:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 365.851887] mask usage: [+|-] ... [ 366.045724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 366.067727] alg: No test for adler32 (adler32-zlib) [ 366.978290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 367.076254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 367.107659] LNet: 9720:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 367.142466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 367.208343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 367.234792] LNet: 9727:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 374.457471] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 08:17:36 (1782217056) [ 374.689719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 374.776606] LNet: 10305:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 374.821558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 374.883214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 382.221483] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 08:17:43 (1782217063) [ 382.308360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 382.358723] LNet: 10884:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 382.406654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 382.543742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 391.388470] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 08:17:52 (1782217072) [ 391.713903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 391.750546] LNet: 11477:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 391.786889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 391.926894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 399.276938] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 08:18:00 (1782217080) [ 399.388085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 399.437533] LNet: 12055:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 399.467983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 399.580662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 399.894967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 407.635757] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 08:18:09 (1782217089) [ 407.838686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 407.871060] LNet: 12651:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 407.909469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 407.980152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 408.630519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 409.484308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 417.604391] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 08:18:18 (1782217098) [ 417.694414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.747776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 417.898694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 426.584545] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 08:18:27 (1782217107) [ 426.810052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 426.829154] LNet: 13833:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 426.840202] LNet: 13833:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 426.900910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 427.005752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 427.046578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 427.783214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 427.851828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 428.483561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 428.515416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 429.052844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 429.145735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 429.786547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 429.823524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 430.375989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 430.429370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 431.308264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 431.345779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 431.914714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 431.971292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 432.622428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 432.692672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 433.488951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 433.562981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 434.290045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 434.379674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 435.120705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 435.159667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 435.565806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 435.634762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 444.931765] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 08:18:45 (1782217125) [ 445.080727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 445.143437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 445.251591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 445.285981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 454.688567] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 08:18:56 (1782217136) [ 455.051721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 455.103254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 455.213515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 455.253316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 463.817340] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 08:19:05 (1782217145) [ 464.140822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.166196] LNet: 15690:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 464.174308] LNet: 15690:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 464.202953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.280976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 464.344271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 471.991588] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 08:19:13 (1782217153) [ 472.061629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 472.128552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 472.231315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 472.261482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 478.940560] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 08:19:20 (1782217160) [ 479.313296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 479.349682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.407717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 479.454958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 486.853581] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 08:19:28 (1782217168) [ 486.955422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 487.023598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 487.223067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 487.299383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 494.625176] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 08:19:36 (1782217176) [ 494.705971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 494.745853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 494.860549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 495.263595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 502.608538] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 08:19:43 (1782217183) [ 502.713122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 502.790716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 503.068876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 503.379932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 511.068735] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 08:19:52 (1782217192) [ 511.178076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 511.240530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 511.400926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 511.766920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 519.267412] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 08:20:01 (1782217201) [ 519.590900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.634990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 520.072762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 520.381753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 528.365962] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 08:20:09 (1782217209) [ 528.455905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 528.494418] LNet: 20510:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 528.500393] LNet: 20510:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 528.537305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 528.656336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 528.709265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 536.730132] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 08:20:18 (1782217218) [ 536.890795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 536.935167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 537.062828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 537.149547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 544.976137] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 08:20:26 (1782217226) [ 545.081254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 545.164940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 545.280179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 552.323316] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 08:20:33 (1782217233) [ 552.458307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 552.499258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 552.606252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 560.271627] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 08:20:41 (1782217241) [ 560.354412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 560.445444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 560.617916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 561.248025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 568.789101] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 08:20:50 (1782217250) [ 569.123867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 569.188923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 569.288892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 569.323582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 577.880205] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 08:20:59 (1782217259) [ 578.211177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 578.248195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 578.355404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 578.391730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 585.099265] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 08:21:06 (1782217266) [ 585.615392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 585.686621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 585.847317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 585.894314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 593.875050] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 08:21:15 (1782217275) [ 594.025672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 594.131179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.348741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 594.411507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 601.847968] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 08:21:23 (1782217283) [ 602.256873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 602.320600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 602.422367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 602.519729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 602.548954] LNetError: 25835:0:(peer.c:2234:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 602.604055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 602.641331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 609.261859] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 08:21:30 (1782217290) [ 609.363644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 609.430660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.547639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.VLYqGK6yr2 [ 609.656542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.VLYqGK6yr2 [ 609.731465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.VLYqGK6yr2 [ 616.696790] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 08:21:38 (1782217298) [ 616.875485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.930074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.978843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 617.019611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 617.075901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 624.417746] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 08:21:45 (1782217305) [ 624.561376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 624.641735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 624.758100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 628.494844] LNet: Message forwarding will be enabled in 2s [ 631.585502] LNet: Message forwarding enabled [ 632.044726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 632.065916] LNet: Message forwarding will stop in 2s to 6s [ 636.736917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 637.795496] LNet: Message forwarding disabled [ 639.510555] LNet: Message forwarding will be enabled in 2s [ 642.912430] LNet: Message forwarding enabled [ 647.359643] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 08:22:08 (1782217328) [ 647.457212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 648.710325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.780486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 655.426403] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 08:22:17 (1782217337) [ 655.574256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 655.641340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 655.743830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 655.778914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 655.800720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 655.829529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 655.848620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 655.873962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 655.901535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 655.922311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 655.942238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 655.974987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 656.007839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 656.057290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 656.089922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 656.135249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 656.172492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 656.210860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 656.249987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 656.297401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 656.329489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 656.349122] LNetError: 28907:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 662.899113] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 08:22:24 (1782217344) [ 663.135957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 663.149373] LNet: 29479:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 663.159726] LNet: 29479:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 14 previous similar messages [ 663.176673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 663.239805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 670.428880] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 08:22:32 (1782217352) [ 672.484926] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg240-server [ 674.622720] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 08:22:36 (1782217356) [ 674.750405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 674.820211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 674.889707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 674.922311] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 676.711873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[41-45]@tcp [ 676.758431] LNet: 30624:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 676.813978] LNetError: 30410:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.41@tcp is being used as a gateway but routing feature is not turned on [ 676.831263] LNetError: 30410:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.41@tcp has gone from up to down [ 677.803534] LNetError: 30410:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.43@tcp is being used as a gateway but routing feature is not turned on [ 677.814647] LNetError: 30410:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 678.827621] LNetError: 30410:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.41@tcp is being used as a gateway but routing feature is not turned on [ 678.854151] LNetError: 30410:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 680.870084] LNetError: 30410:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.45@tcp is being used as a gateway but routing feature is not turned on [ 680.895852] LNetError: 30410:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 684.333893] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 08:22:45 (1782217365) [ 684.959880] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.41@tcp [ 684.971897] LNetError: 30410:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.45@tcp is being used as a gateway but routing feature is not turned on [ 684.980572] LNetError: 30410:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 18 previous similar messages [ 686.340195] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg240-server [ 688.330725] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 08:22:49 (1782217369) [ 688.456334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 688.492347] LNetError: 31553:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 688.523186] LNet: Removed LNI 192.168.202.40@tcp [ 688.575886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.726705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 688.759038] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 689.633464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[41-46/2]@tcp [ 689.655983] LNet: 31677:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 689.661375] LNet: 31677:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 689.671940] LNetError: 31570:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.41@tcp has gone from up to down [ 689.687107] LNetError: 31570:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 689.702364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[41-46/2]@tcp [ 697.044984] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 08:22:58 (1782217378) [ 697.385246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.410240] LNetError: 32272:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 697.447480] LNet: Removed LNI 192.168.202.40@tcp [ 697.474764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.544579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 697.604140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.640680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.721793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 697.792299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.844352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.899511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 698.002652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.038799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.116752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 698.229231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.282862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.334352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 698.421547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.470228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.526429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 698.613748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.656106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.723831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 698.813697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.866980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.940603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 699.046773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 699.090714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.185329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 699.283559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 699.313309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.369991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 699.434430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 699.473963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.518444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 699.567643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 699.619384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.710682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 706.609327] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 08:23:08 (1782217388) [ 708.936474] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg240-server [ 710.944360] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 08:23:12 (1782217392) [ 714.092656] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg240-server [ 717.106514] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 08:23:17 (1782217397) [ 718.349417] Key type .llcrypt unregistered [ 718.351644] Key type ._llcrypt unregistered [ 728.595060] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 728.806215] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 728.816136] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 728.821110] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 731.552253] Key type ._llcrypt registered [ 731.555823] Key type .llcrypt registered [ 731.599132] LNet: 33947:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 731.599132] mask usage: [+|-] ... [ 731.870654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 731.882306] alg: No test for adler32 (adler32-zlib) [ 732.881488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 732.963792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 733.012417] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 734.008071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 734.045915] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 736.247527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 736.265286] LNet: Removed LNI 10.1.2.3@tcp [ 736.351033] LNet: 34391:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 736.353478] LNetError: 34391:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 736.364248] LNet: Removed LNI 192.168.202.40@tcp [ 737.050506] Key type .llcrypt unregistered [ 737.053970] Key type ._llcrypt unregistered [ 750.293660] Key type ._llcrypt registered [ 750.296215] Key type .llcrypt registered [ 751.831872] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 08:23:53 (1782217433) [ 752.672520] Key type .llcrypt unregistered [ 752.673975] Key type ._llcrypt unregistered [ 761.043648] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 761.178909] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 761.186553] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 761.194854] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 763.895253] Key type ._llcrypt registered [ 763.901288] Key type .llcrypt registered [ 763.975925] LNet: 35428:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 763.975925] mask usage: [+|-] ... [ 764.213904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 764.231592] alg: No test for adler32 (adler32-zlib) [ 765.124565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 765.168258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 765.190427] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 766.119244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 766.155552] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 769.154726] LNet: 35886:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 769.168613] LNetError: 35886:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 769.180362] LNet: Removed LNI 10.1.2.3@tcp [ 769.213669] LNet: Removed LNI 192.168.202.40@tcp [ 770.149655] Key type .llcrypt unregistered [ 770.156389] Key type ._llcrypt unregistered [ 783.908567] Key type ._llcrypt registered [ 783.910445] Key type .llcrypt registered [ 785.161407] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 08:24:26 (1782217466) [ 786.020629] Key type .llcrypt unregistered [ 786.027614] Key type ._llcrypt unregistered [ 793.550804] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 793.820547] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 793.831665] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 793.843686] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 796.447718] Key type ._llcrypt registered [ 796.450627] Key type .llcrypt registered [ 796.479721] LNet: 36921:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 796.479721] mask usage: [+|-] ... [ 796.624553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 796.643293] alg: No test for adler32 (adler32-zlib) [ 797.623141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 797.678967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 797.733456] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 799.665111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 799.679884] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 802.170312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 802.210344] LNet: Removed LNI 10.1.2.3@tcp [ 802.237986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 802.269564] LNet: Removed LNI 10.1.2.31@tcp [ 802.284783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 802.313903] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 803.087313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 803.104600] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 804.114703] LNet: 37805:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 804.123234] LNetError: 37805:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 804.130414] LNet: Removed LNI 10.1.2.31@tcp [ 804.145777] LNet: Removed LNI 10.1.2.3@tcp [ 804.901528] Key type .llcrypt unregistered [ 804.906784] Key type ._llcrypt unregistered [ 816.996989] Key type ._llcrypt registered [ 816.999704] Key type .llcrypt registered [ 818.998685] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 08:24:59 (1782217499) [ 819.258563] LNet: 38598:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 819.258563] mask usage: [+|-] ... [ 819.460619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 819.483444] alg: No test for adler32 (adler32-zlib) [ 820.438297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.476872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 820.492423] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 821.414362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 821.441845] LNet: 38758:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 821.451944] LNetError: 38758:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 821.463945] LNet: Removed LNI 192.168.202.40@tcp [ 821.480902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 821.529620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.40@tcp [ 821.567985] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 822.365434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.40@tcp [ 822.392853] LNet: Removed LNI 192.168.202.40@tcp [ 822.466625] LNet: 38893:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 823.122320] Key type .llcrypt unregistered [ 823.124864] Key type ._llcrypt unregistered [ 835.507852] Key type ._llcrypt registered [ 835.519092] Key type .llcrypt registered [ 836.935264] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 08:25:18 (1782217518) [ 839.016977] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg240-server [ 840.965908] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 08:25:22 (1782217522) [ 841.521321] Key type .llcrypt unregistered [ 841.523063] Key type ._llcrypt unregistered [ 848.701889] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 848.810746] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 848.828786] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 848.840697] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 851.847099] Key type ._llcrypt registered [ 851.853208] Key type .llcrypt registered [ 851.897430] LNet: 40273:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 851.897430] mask usage: [+|-] ... [ 852.061754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 852.083403] alg: No test for adler32 (adler32-zlib) [ 853.108466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 853.165026] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 854.050724] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 856.381734] LNet: 40733:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 856.387813] LNetError: 40733:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 856.392852] LNet: Removed LNI 192.168.202.40@tcp1 [ 856.403328] LNet: Removed LNI 10.1.2.3@tcp2 [ 857.096351] Key type .llcrypt unregistered [ 857.099705] Key type ._llcrypt unregistered [ 870.630930] Key type ._llcrypt registered [ 870.632953] Key type .llcrypt registered [ 872.253178] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 08:25:53 (1782217553) [ 872.524693] LNet: 41536:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 872.524693] mask usage: [+|-] ... [ 872.731904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 872.744984] alg: No test for adler32 (adler32-zlib) [ 873.873899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 873.988704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 874.020099] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 874.952909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 874.975497] LNet: 41702:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 874.982625] LNetError: 41702:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 874.992930] LNet: Removed LNI 192.168.202.40@tcp [ 875.016551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 875.095653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 875.122581] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 882.871486] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 08:26:04 (1782217564) [ 883.110593] LNet: 42294:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.130737] LNetError: 42294:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.169087] LNet: Removed LNI 192.168.202.40@tcp [ 883.834557] Key type .llcrypt unregistered [ 883.837214] Key type ._llcrypt unregistered [ 890.265974] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 890.523499] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 890.535700] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 890.541554] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 892.177955] Key type ._llcrypt registered [ 892.182270] Key type .llcrypt registered [ 892.218658] LNet: 42566:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 892.218658] mask usage: [+|-] ... [ 892.388493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 892.414635] alg: No test for adler32 (adler32-zlib) [ 893.368538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 893.410815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 893.435861] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 894.390765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 894.424294] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 897.406162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 897.435567] LNet: 43032:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 897.447314] LNetError: 43032:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 897.455773] LNet: Removed LNI 10.1.2.3@tcp [ 897.474348] LNet: Removed LNI 192.168.202.40@tcp [ 897.506785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 897.638701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 897.680221] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 897.683852] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 898.410884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 898.466384] LNet: 43059:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 898.471738] LNetError: 43059:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 898.475884] LNetError: 43059:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 898.479816] LNet: Removed LNI 10.1.2.3@tcp [ 898.498394] LNet: Removed LNI 192.168.202.40@tcp [ 898.509883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 898.590455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 898.646990] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 898.655873] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 899.545546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 899.587076] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 900.607945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 900.631900] LNet: 43222:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.639941] LNetError: 43222:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 900.648105] LNetError: 43222:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 900.657937] LNet: Removed LNI 10.1.2.3@tcp [ 900.667504] LNet: Removed LNI 192.168.202.40@tcp [ 900.680744] LNet: Removed LNI 192.168.202.40@tcp2 [ 900.696599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 900.745623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 900.768573] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 900.773712] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 900.778247] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 908.869868] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 08:26:30 (1782217590) [ 909.111621] LNet: 43818:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.121510] LNetError: 43818:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 909.134807] LNetError: 43818:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 909.146077] LNet: Removed LNI 10.1.2.3@tcp [ 909.147598] LNet: Removed LNI 192.168.202.40@tcp [ 909.161120] LNet: Removed LNI 192.168.202.40@tcp2 [ 910.002342] Key type .llcrypt unregistered [ 910.003769] Key type ._llcrypt unregistered [ 918.232808] Key type ._llcrypt registered [ 918.236362] Key type .llcrypt registered [ 918.302218] LNet: 44046:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 918.302218] mask usage: [+|-] ... [ 918.581282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 918.613531] alg: No test for adler32 (adler32-zlib) [ 919.611856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 919.703375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 919.739105] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 920.817496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 920.840677] LNet: 44215:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.844273] LNetError: 44215:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 920.856561] LNet: Removed LNI 192.168.202.40@tcp [ 920.939901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 921.040720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 921.128149] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 922.140735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 922.162367] LNet: 44246:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 922.167278] LNetError: 44246:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 922.181779] LNet: Removed LNI 192.168.202.40@tcp [ 922.211496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 922.320387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 922.369391] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 923.040766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 923.073165] LNet: Removed LNI 192.168.202.40@tcp [ 923.090697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 923.154340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 923.182121] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 923.914060] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 924.030261] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 924.035521] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 924.042702] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 926.690440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 926.728519] LNet: 44356:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.735980] LNet: 44356:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 926.739918] LNetError: 44356:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 926.745217] LNetError: 44356:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 926.779570] LNet: Removed LNI 192.168.202.40@tcp [ 926.804312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 926.855994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 926.887359] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 935.204817] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 08:26:56 (1782217616) [ 935.436345] LNet: 44953:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 935.441234] LNetError: 44953:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 935.461300] LNet: Removed LNI 192.168.202.40@tcp [ 936.259508] Key type .llcrypt unregistered [ 936.261667] Key type ._llcrypt unregistered [ 944.601417] Key type ._llcrypt registered [ 944.602948] Key type .llcrypt registered [ 944.648899] LNet: 45179:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 944.648899] mask usage: [+|-] ... [ 944.871139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 944.905311] alg: No test for adler32 (adler32-zlib) [ 945.961076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 946.034598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 946.064469] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 947.178654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 947.198132] LNet: 45353:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 947.202907] LNetError: 45353:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 947.212804] LNet: Removed LNI 192.168.202.40@tcp [ 947.249702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 947.348859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 947.389415] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 954.924765] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 08:27:16 (1782217636) [ 955.077301] LNet: 45938:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 955.093747] LNetError: 45938:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 955.118468] LNet: Removed LNI 192.168.202.40@tcp [ 955.961460] Key type .llcrypt unregistered [ 955.964389] Key type ._llcrypt unregistered [ 964.095456] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 964.539660] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 964.562555] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 964.571137] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 966.305968] Key type ._llcrypt registered [ 966.307916] Key type .llcrypt registered [ 966.376151] LNet: 46214:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 966.376151] mask usage: [+|-] ... [ 966.621600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 966.634169] alg: No test for adler32 (adler32-zlib) [ 967.631231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 967.716713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 967.762440] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 969.056119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 969.073650] LNet: 46388:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 969.079071] LNetError: 46388:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 969.094584] LNet: Removed LNI 192.168.202.40@tcp [ 969.117348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 969.183901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 969.212837] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 977.144478] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 08:27:38 (1782217658) [ 977.385239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 977.442462] LNet: 46986:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 977.452459] LNetError: 46986:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 977.473079] LNet: Removed LNI 192.168.202.40@tcp [ 977.514347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 977.620263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 985.484414] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 08:27:46 (1782217666) [ 985.699430] LNet: 47565:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 986.420972] Key type .llcrypt unregistered [ 986.423525] Key type ._llcrypt unregistered [ 994.321613] Key type ._llcrypt registered [ 994.325689] Key type .llcrypt registered [ 994.391873] LNet: 47795:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 994.391873] mask usage: [+|-] ... [ 994.599178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 994.613715] alg: No test for adler32 (adler32-zlib) [ 995.606517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 995.685495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 995.744191] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 997.322789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 997.372760] LNet: 47967:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 997.384113] LNetError: 47967:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 997.413757] LNet: Removed LNI 192.168.202.40@tcp [ 997.452462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 997.509840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 997.536299] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1005.506083] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 08:28:06 (1782217686) [ 1005.658192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1005.726495] LNet: 48557:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1005.744228] LNetError: 48557:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1005.775465] LNet: Removed LNI 192.168.202.40@tcp [ 1005.823504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1005.910300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1013.204429] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 08:28:14 (1782217694) [ 1015.225706] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1017.888851] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 08:28:18 (1782217698) [ 1019.903638] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1021.534682] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 08:28:23 (1782217703) [ 1023.494535] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1025.376902] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 08:28:26 (1782217706) [ 1027.057424] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1029.110501] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 08:28:30 (1782217710) [ 1031.496432] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1033.281232] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 08:28:35 (1782217715) [ 1033.881031] LNet: 50615:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1034.630600] Key type .llcrypt unregistered [ 1034.634184] Key type ._llcrypt unregistered [ 1042.456347] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1042.579957] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1042.589639] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1042.592038] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1045.393706] Key type ._llcrypt registered [ 1045.396148] Key type .llcrypt registered [ 1045.425479] LNet: 50901:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1045.425479] mask usage: [+|-] ... [ 1045.630046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1045.660133] alg: No test for adler32 (adler32-zlib) [ 1046.655851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1046.730551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1046.748891] LNet: Added LNI 192.168.202.40@tcp [8/512/0/180] [ 1046.752821] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1046.754962] LNet: 50939:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1046.937763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1047.017486] LNet: 50958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1047.032751] LNetError: 50958:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1047.046501] LNet: Removed LNI 10.1.2.3@tcp [ 1047.049593] LNetError: 50958:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1047.071990] LNet: Removed LNI 192.168.202.40@tcp [ 1047.096879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1047.150703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1047.170816] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1047.233938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1047.249749] LNet: 50982:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1047.254771] LNetError: 50982:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1047.265920] LNet: Removed LNI 192.168.202.40@tcp [ 1047.283176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1047.348468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1047.370195] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1047.375435] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1047.378513] LNet: 50990:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1047.470737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1047.498276] LNet: 51008:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1047.505496] LNetError: 51008:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1047.508827] LNet: Removed LNI 10.1.2.3@tcp [ 1047.517698] LNetError: 51008:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1047.534519] LNet: Removed LNI 192.168.202.40@tcp [ 1047.562633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1047.605768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1047.625153] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1047.632202] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1047.634889] LNet: 51016:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1047.788114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1047.803588] LNet: 51042:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1047.811078] LNetError: 51042:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1047.813620] LNet: Removed LNI 10.1.2.3@tcp [ 1047.819630] LNetError: 51042:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1047.835989] LNet: Removed LNI 192.168.202.40@tcp [ 1047.851843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1047.959601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1048.011762] LNet: Added LNI 192.168.202.40@tcp [8/256/0/182] [ 1048.022486] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1048.162747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1048.196613] LNet: 51070:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1048.201965] LNetError: 51070:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1048.211941] LNet: Removed LNI 10.1.2.3@tcp [ 1048.222040] LNetError: 51070:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1048.277467] LNet: Removed LNI 192.168.202.40@tcp [ 1048.307282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1048.426196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1048.486271] LNet: Added LNI 192.168.202.40@tcp [8/256/0/182] [ 1048.498929] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1048.502863] LNet: 51078:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1048.587914] LNet: Removed LNI 10.1.2.3@tcp [ 1048.604585] LNet: Removed LNI 192.168.202.40@tcp [ 1049.287660] Key type .llcrypt unregistered [ 1049.295685] Key type ._llcrypt unregistered [ 1062.364933] Key type ._llcrypt registered [ 1062.367471] Key type .llcrypt registered [ 1063.879743] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 08:29:05 (1782217745) [ 1064.092347] LNet: 51905:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1064.092347] mask usage: [+|-] ... [ 1064.292803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1064.309263] alg: No test for adler32 (adler32-zlib) [ 1065.376668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1065.457708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1065.483322] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1066.580080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.40 [ 1066.614768] LNet: 52069:0:(config.c:1627:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1066.630549] LNet: Removed LNI 192.168.202.40@tcp [ 1074.002747] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 08:29:15 (1782217755) [ 1074.186535] LNet: 52632:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1074.196253] LNetError: 52632:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1074.239106] LNet: Removed LNI 192.168.202.40@tcp [ 1074.957207] Key type .llcrypt unregistered [ 1074.961231] Key type ._llcrypt unregistered [ 1082.078594] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1082.411396] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1085.438766] Key type ._llcrypt registered [ 1085.442235] Key type .llcrypt registered [ 1085.497950] LNet: 52922:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1085.497950] mask usage: [+|-] ... [ 1085.723612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1085.733623] alg: No test for adler32 (adler32-zlib) [ 1087.163164] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1094.894600] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 08:29:36 (1782217776) [ 1095.015821] LNet: 53545:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1095.022198] LNetError: 53545:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1095.054409] LNet: Removed LNI 10.1.2.3@tcp [ 1095.790812] Key type .llcrypt unregistered [ 1095.793602] Key type ._llcrypt unregistered [ 1103.848845] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1104.290040] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1104.299116] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1104.303490] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1106.182766] Key type ._llcrypt registered [ 1106.186223] Key type .llcrypt registered [ 1106.280207] LNet: 53820:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1106.280207] mask usage: [+|-] ... [ 1106.458879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1106.479888] alg: No test for adler32 (adler32-zlib) [ 1107.518146] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1115.655166] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 08:29:57 (1782217797) [ 1117.529556] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1119.888227] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 08:30:01 (1782217801) [ 1120.016492] LNet: 54725:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1120.023961] LNetError: 54725:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1120.052369] LNet: Removed LNI 10.1.2.3@tcp [ 1120.740631] Key type .llcrypt unregistered [ 1120.742294] Key type ._llcrypt unregistered [ 1129.200025] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1129.490163] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1129.492096] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1129.494729] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1132.333952] Key type ._llcrypt registered [ 1132.335314] Key type .llcrypt registered [ 1132.401168] LNet: 55008:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1132.401168] mask usage: [+|-] ... [ 1132.604554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1132.617065] alg: No test for adler32 (adler32-zlib) [ 1133.652446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1133.755101] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1144.662272] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 08:30:26 (1782217826) [ 1145.299130] LNet: 55964:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1145.304673] LNetError: 55964:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1145.317366] LNet: Removed LNI 10.1.2.3@tcp [ 1146.154340] Key type .llcrypt unregistered [ 1146.156506] Key type ._llcrypt unregistered [ 1153.610424] Key type ._llcrypt registered [ 1153.615227] Key type .llcrypt registered [ 1153.675186] LNet: 56188:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1153.675186] mask usage: [+|-] ... [ 1154.025791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1154.057804] alg: No test for adler32 (adler32-zlib) [ 1155.101472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1155.160920] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1166.253963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.140@tcp [ 1166.957458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_interrupt -r 1 [ 1167.034901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_interrupt -r 1 [ 1167.082636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 1167.105685] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1167.294945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1167.321996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1167.351960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1167.387190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1167.426101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1167.453647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1167.463791] LNet: 1 local NIs in recovery (showing 1): 192.168.202.40@tcp [ 1167.481231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1167.499152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1167.512582] LNet: 56404:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1167.529724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1167.571651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1167.599946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1167.622587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1167.640804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1167.673222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1167.708424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1168.307427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_dropped -r 1 [ 1168.357394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_dropped -r 1 [ 1168.437594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 1168.497788] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1168.826458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1168.892147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1168.976359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1169.001658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1169.042537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1169.077079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1169.111376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1169.174728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1169.218572] LNet: 56513:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1169.249601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1169.313701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1169.363655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1169.427824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1169.486746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1169.517950] LNet: There was an unexpected network error while writing to 192.168.202.40: rc = -22 [ 1169.598347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1169.660815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1170.447459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_aborted -r 1 [ 1170.515072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_aborted -r 1 [ 1170.581509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 1170.963549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1170.997936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1171.057915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1171.101101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1171.134969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1171.215407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1171.281634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1171.342279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1171.376283] LNet: 56622:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1171.406944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1171.471292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1171.528779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1171.591074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1171.693316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1171.762958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1171.842149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1172.999743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_no_route -r 1 [ 1173.159585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_no_route -r 1 [ 1173.247642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 1173.313210] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1173.322475] LNet: Skipped 1 previous similar message [ 1173.782977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1173.842104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1173.892549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1173.930775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1173.979914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1174.035703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1174.069218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1174.106640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1174.141288] LNet: 56732:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1174.173261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1174.230143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1174.284446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1174.351665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1174.383741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1174.411934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1174.437711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1175.064286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_timeout -r 1 [ 1175.107395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_timeout -r 1 [ 1175.171331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 1175.615297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1175.690715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1175.791536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1175.858195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1175.914504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1175.976906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1176.014621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1176.058124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1176.094418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1176.167678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1176.202405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1176.236650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1176.278654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1176.315207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1176.349874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1177.082839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_error -r 1 [ 1177.145955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_error -r 1 [ 1177.200203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 1177.591872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1177.628713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1177.655038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1177.686554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1177.767467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1177.825387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1177.897682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1177.944464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1178.013949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1178.073655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1178.119085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1178.147577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1178.173842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1178.209994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1178.248546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1178.531552] LNet: 56992:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1178.569628] LNetError: 56992:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1178.612408] LNet: Removed LNI 192.168.202.40@tcp [ 1179.172650] Key type .llcrypt unregistered [ 1179.175852] Key type ._llcrypt unregistered [ 1200.397613] Key type ._llcrypt registered [ 1200.400352] Key type .llcrypt registered [ 1202.334834] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 08:31:23 (1782217883) [ 1203.645903] Key type .llcrypt unregistered [ 1203.650717] Key type ._llcrypt unregistered [ 1210.326868] Key type ._llcrypt registered [ 1210.333763] Key type .llcrypt registered [ 1210.400296] LNet: 58031:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1210.400296] mask usage: [+|-] ... [ 1210.695893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1210.755282] alg: No test for adler32 (adler32-zlib) [ 1211.736948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1211.784148] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1220.568883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.140@tcp [ 1232.289799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1232.347648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1232.386473] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 1234.354112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_interrupt -r 1 [ 1234.422457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e local_interrupt -r 1 [ 1234.497743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_interrupt -r 1 [ 1234.537864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e local_interrupt -r 1 [ 1234.590082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e local_interrupt -r 1 [ 1234.671766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e local_interrupt -r 1 [ 1234.751598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e local_interrupt -r 1 [ 1234.796789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e local_interrupt -r 1 [ 1234.851493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 1234.880200] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1234.887471] LNet: 1 local NIs in recovery (showing 1): 192.168.202.40@tcp [ 1235.197532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1235.243718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1235.338971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1235.407330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1235.468450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1235.551821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1235.607842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1235.655830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1235.673540] LNet: 58462:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1235.696790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1235.728664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1235.758406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1235.811298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1235.852603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1235.904406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1235.936761] LNet: There was an unexpected network error while writing to 192.168.202.40: rc = -22 [ 1235.955443] LNet: Skipped 3 previous similar messages [ 1235.983247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1236.892148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_dropped -r 1 [ 1236.938131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e local_dropped -r 1 [ 1237.005212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_dropped -r 1 [ 1237.063351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e local_dropped -r 1 [ 1237.151255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e local_dropped -r 1 [ 1237.190401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e local_dropped -r 1 [ 1237.287924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e local_dropped -r 1 [ 1237.358295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e local_dropped -r 1 [ 1237.421535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 1237.443807] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1237.791280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1237.852769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1237.902470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1237.982923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1238.043402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1238.110973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1238.171197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1238.242790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1238.288740] LNet: 58584:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1238.327574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1238.387267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1238.456736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1238.522533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1238.585456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1238.640681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1238.676610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1239.346745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_aborted -r 1 [ 1239.383671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e local_aborted -r 1 [ 1239.421214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_aborted -r 1 [ 1239.465987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e local_aborted -r 1 [ 1239.526616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e local_aborted -r 1 [ 1239.573914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e local_aborted -r 1 [ 1239.619503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e local_aborted -r 1 [ 1239.656240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e local_aborted -r 1 [ 1239.713287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 1239.773594] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1239.799811] LNet: Skipped 2 previous similar messages [ 1240.121222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1240.178924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1240.212706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1240.280491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1240.342345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1240.414778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1240.467232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1240.592069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1240.636519] LNet: 58705:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1240.669538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1240.756058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1240.862285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1240.930311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1240.977765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1241.015992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1241.081709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1241.761429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_no_route -r 1 [ 1241.809261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e local_no_route -r 1 [ 1241.845479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_no_route -r 1 [ 1241.875996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e local_no_route -r 1 [ 1241.914621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e local_no_route -r 1 [ 1241.959994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e local_no_route -r 1 [ 1242.006565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e local_no_route -r 1 [ 1242.070108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e local_no_route -r 1 [ 1242.134163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 1242.590913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1242.636687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1242.689641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1242.723946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1242.802833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1242.871448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1242.937114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1243.019304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1243.063498] LNet: 58826:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1243.098620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1243.174953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1243.276409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1243.305551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1243.387521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1243.444549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1243.510442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1244.274288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_timeout -r 1 [ 1244.350238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e local_timeout -r 1 [ 1244.435494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_timeout -r 1 [ 1244.513282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e local_timeout -r 1 [ 1244.593464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e local_timeout -r 1 [ 1244.633064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e local_timeout -r 1 [ 1244.718530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e local_timeout -r 1 [ 1244.793764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e local_timeout -r 1 [ 1244.864218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 1244.890619] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1244.900202] LNet: Skipped 5 previous similar messages [ 1245.200650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1245.247736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1245.287345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1245.329245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1245.377958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1245.421542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1245.472846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1245.515332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1245.552673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1245.586601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1245.624252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1245.660686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1245.696338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1245.737950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1245.791392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1246.406721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_error -r 1 [ 1246.479523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e local_error -r 1 [ 1246.609833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_error -r 1 [ 1246.708258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e local_error -r 1 [ 1246.806209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e local_error -r 1 [ 1246.932637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e local_error -r 1 [ 1247.002827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e local_error -r 1 [ 1247.079943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e local_error -r 1 [ 1247.166978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 1247.617966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1247.682640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1247.725566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1247.760404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1247.803701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1247.841422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1247.882204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1247.944688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1247.978142] LNet: 59069:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1247.984118] LNet: 59069:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1248.040963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1248.106220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1248.138599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1248.170753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1248.201200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1248.233822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1248.265330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1253.156803] LNet: 59145:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1253.166914] LNetError: 59145:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1253.170938] LNet: Removed LNI 192.168.202.40@tcp [ 1254.183534] LNet: Removed LNI 192.168.202.40@tcp1 [ 1254.468600] Key type .llcrypt unregistered [ 1254.473225] Key type ._llcrypt unregistered [ 1273.679763] Key type ._llcrypt registered [ 1273.682428] Key type .llcrypt registered [ 1275.029159] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 08:32:36 (1782217956) [ 1275.812627] Key type .llcrypt unregistered [ 1275.814830] Key type ._llcrypt unregistered [ 1283.310912] Key type ._llcrypt registered [ 1283.313602] Key type .llcrypt registered [ 1283.339655] LNet: 60181:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1283.339655] mask usage: [+|-] ... [ 1283.503258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1283.513956] alg: No test for adler32 (adler32-zlib) [ 1284.554843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1284.602368] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1292.377495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.140@tcp [ 1293.252313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e remote_dropped -r 1 [ 1293.320651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e remote_dropped -r 1 [ 1293.371612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 1293.440198] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1293.821392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1293.852289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1293.882099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1293.904929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1293.948341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1294.000667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1294.060300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1294.126789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1294.151945] LNet: 60396:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1294.171944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1294.218640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1294.296678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1294.355712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1294.398735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1294.460333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1294.527781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1295.102475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e remote_error -r 1 [ 1295.151332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e remote_error -r 1 [ 1295.183368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 1295.220869] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1295.507300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1295.565865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1295.659261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1295.730700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1295.812592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1295.969700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1296.025057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1296.090586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1296.137780] LNet: 60506:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1296.190424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1296.272500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1296.345705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1296.403158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1296.471880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1296.533505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1296.597738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1297.420656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e remote_timeout -r 1 [ 1297.469509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e remote_timeout -r 1 [ 1297.534502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 1297.576703] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1297.946344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1298.061411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1298.190163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1298.255945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1298.332312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1298.412670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1298.467683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1298.542047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1298.564056] LNet: 60615:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1298.587872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1298.646950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1298.712200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1298.757411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1298.812808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1298.861645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1298.924373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1299.334452] LNet: 60656:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1299.349144] LNetError: 60656:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1299.392677] LNet: Removed LNI 192.168.202.40@tcp [ 1300.132483] Key type .llcrypt unregistered [ 1300.136232] Key type ._llcrypt unregistered [ 1322.193074] Key type ._llcrypt registered [ 1322.194449] Key type .llcrypt registered [ 1323.542756] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 08:33:25 (1782218005) [ 1324.425386] Key type .llcrypt unregistered [ 1324.427911] Key type ._llcrypt unregistered [ 1331.546930] Key type ._llcrypt registered [ 1331.549896] Key type .llcrypt registered [ 1331.599262] LNet: 61696:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1331.599262] mask usage: [+|-] ... [ 1331.725438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1331.749179] alg: No test for adler32 (adler32-zlib) [ 1332.746823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1332.776402] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1341.488936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.140@tcp [ 1350.790076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1350.842754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1350.867577] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 1352.678455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e remote_dropped -r 1 [ 1352.734158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e remote_dropped -r 1 [ 1352.820558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e remote_dropped -r 1 [ 1352.893658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e remote_dropped -r 1 [ 1352.947971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e remote_dropped -r 1 [ 1353.018138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e remote_dropped -r 1 [ 1353.093969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e remote_dropped -r 1 [ 1353.171645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e remote_dropped -r 1 [ 1353.214788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 1353.241143] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1353.244395] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.140@tcp [ 1353.515294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1353.565394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1353.623579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1353.674195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1353.709471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1353.746686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1353.796511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1353.848369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1353.900617] LNet: 62126:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1353.931706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1353.983168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1354.041951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1354.099986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1354.138455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1354.198733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1354.269127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1354.345854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1355.147397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e remote_error -r 1 [ 1355.183710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e remote_error -r 1 [ 1355.223083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e remote_error -r 1 [ 1355.249723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e remote_error -r 1 [ 1355.276468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e remote_error -r 1 [ 1355.313413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e remote_error -r 1 [ 1355.348453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e remote_error -r 1 [ 1355.378713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e remote_error -r 1 [ 1355.407950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 1355.427314] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1355.432302] LNet: Skipped 2 previous similar messages [ 1355.634375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1355.660139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1355.682943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1355.712627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1355.737697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1355.763719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1355.793455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1355.820493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1355.843173] LNet: 62250:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1355.858585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1355.898160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1355.927938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1355.965419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1356.008505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1356.071124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1356.123538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1356.182984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1356.703397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e remote_timeout -r 1 [ 1356.744981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e remote_timeout -r 1 [ 1356.782742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e remote_timeout -r 1 [ 1356.886237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e remote_timeout -r 1 [ 1356.927992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e remote_timeout -r 1 [ 1356.963866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e remote_timeout -r 1 [ 1357.056167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e remote_timeout -r 1 [ 1357.106406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e remote_timeout -r 1 [ 1357.132529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.140@tcp [ 1357.143401] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1357.285678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1357.299885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1357.313190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1357.332554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1357.350656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1357.367952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1357.390458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1357.413141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1357.426614] LNet: 62373:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1357.438907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1357.460930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1357.484901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1357.504514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1357.522284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1357.544494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1357.568315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1357.594597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1362.648659] LNet: 62451:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1362.657381] LNetError: 62451:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1362.668556] LNet: Removed LNI 192.168.202.40@tcp [ 1362.688183] LNet: Removed LNI 192.168.202.40@tcp1 [ 1363.359494] Key type .llcrypt unregistered [ 1363.369535] Key type ._llcrypt unregistered [ 1383.897812] Key type ._llcrypt registered [ 1383.898710] Key type .llcrypt registered [ 1385.060286] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 08:34:26 (1782218066) [ 1386.814670] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1388.884170] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 08:34:30 (1782218070) [ 1389.770300] Key type .llcrypt unregistered [ 1389.775714] Key type ._llcrypt unregistered [ 1396.181980] Key type ._llcrypt registered [ 1396.186317] Key type .llcrypt registered [ 1396.247781] LNet: 63783:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1396.247781] mask usage: [+|-] ... [ 1396.523597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1396.564956] alg: No test for adler32 (adler32-zlib) [ 1397.549217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1397.573548] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1405.173685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.140@tcp [ 1405.865709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e network_timeout -r 1 [ 1405.901254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e network_timeout -r 1 [ 1405.933585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.140@tcp [ 1405.954477] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1406.195695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1406.227532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1406.256911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1406.282334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1406.308724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1406.332930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1406.361660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1406.390328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1406.404415] LNet: 63998:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1406.417843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1406.458570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1406.494946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1406.523527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1406.551238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1406.578748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1406.829939] LNet: 64037:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1406.835911] LNetError: 64037:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1407.912106] LNet: Removed LNI 192.168.202.40@tcp [ 1408.358591] Key type .llcrypt unregistered [ 1408.362655] Key type ._llcrypt unregistered [ 1428.685974] Key type ._llcrypt registered [ 1428.691244] Key type .llcrypt registered [ 1428.735592] LNet: 64502:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1428.735592] mask usage: [+|-] ... [ 1429.051250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1429.095241] alg: No test for adler32 (adler32-zlib) [ 1430.100165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1430.141622] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1438.345882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.140@tcp [ 1449.041187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1449.118466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1449.162408] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 1450.848947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e network_timeout -r 1 [ 1450.890711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e network_timeout -r 1 [ 1450.944138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e network_timeout -r 1 [ 1451.001449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e network_timeout -r 1 [ 1451.041689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e network_timeout -r 1 [ 1451.089883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e network_timeout -r 1 [ 1451.158544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e network_timeout -r 1 [ 1451.216335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e network_timeout -r 1 [ 1451.281974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.140@tcp [ 1451.313478] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1451.583117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1451.626607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1451.679098] LNet: 1 local NIs in recovery (showing 1): 192.168.202.40@tcp [ 1451.681705] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.140@tcp [ 1451.684721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1451.746334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1451.776990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1451.843958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1451.894534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1451.936552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1451.960843] LNet: 64932:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1451.991797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1452.043866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1452.090214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1452.118423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1452.159481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1452.186332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1456.553856] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1456.903250] LNet: 65006:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1456.908260] LNetError: 65006:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1456.918137] LNet: Removed LNI 192.168.202.40@tcp [ 1456.941855] LNet: Removed LNI 192.168.202.40@tcp1 [ 1457.336531] Key type .llcrypt unregistered [ 1457.342455] Key type ._llcrypt unregistered [ 1475.649493] Key type ._llcrypt registered [ 1475.652858] Key type .llcrypt registered [ 1476.971086] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 08:35:58 (1782218158) [ 1477.267304] LNet: 65808:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1477.267304] mask usage: [+|-] ... [ 1477.465751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1477.481342] alg: No test for adler32 (adler32-zlib) [ 1478.418551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1478.461860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1478.480609] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1479.204748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1479.223702] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 1480.135243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1480.198243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1480.243990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1480.275170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1480.302258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1480.344238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1480.380779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.40@tcp [ 1480.410000] LNetError: 66108:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.202.40@tcp to recovery [ 1480.481630] LNet: 1 local NIs in recovery (showing 1): 192.168.202.40@tcp [ 1480.481937] LNet: There was an unexpected network error while writing to 192.168.202.40: rc = -22 [ 1482.530287] LNet: There was an unexpected network error while writing to 192.168.202.40: rc = -22 [ 1482.790849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1486.623534] LNet: There was an unexpected network error while writing to 192.168.202.40: rc = -22 [ 1487.391355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1487.421896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1487.472420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1487.487992] LNet: 66215:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1487.497299] LNetError: 66215:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1488.543637] LNet: Removed LNI 192.168.202.40@tcp [ 1488.547512] LNetError: 66215:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1488.559966] LNet: Removed LNI 192.168.202.40@tcp1 [ 1488.577478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1488.614851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1488.652447] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1489.319588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1489.347510] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 1490.090495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1490.149134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1490.201498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1490.233778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1490.261342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1490.292231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1490.318615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1490.357916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.40@tcp [ 1490.378496] LNetError: 66488:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.202.40@tcp to recovery [ 1490.655464] LNet: There was an unexpected network error while writing to 192.168.202.40: rc = -22 [ 1493.879572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1496.799873] LNet: There was an unexpected network error while writing to 192.168.202.40: rc = -22 [ 1496.804886] LNet: Skipped 1 previous similar message [ 1501.940845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1501.970582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1502.008201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1508.938548] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 08:36:30 (1782218190) [ 1509.201554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1509.230723] LNet: 67206:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1509.237218] LNetError: 67206:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1509.245223] LNet: Removed LNI 192.168.202.40@tcp [ 1509.263156] LNet: Removed LNI 192.168.202.40@tcp1 [ 1509.295471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1509.354928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1509.372736] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1510.206996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1510.235268] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 1511.158843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1511.209288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1511.233473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1511.261782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1511.294551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1511.338661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1511.376705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.40@tcp --health 0 [ 1511.394414] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.40@tcp [ 1511.471526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1512.415896] LNet: There was an unexpected network error while writing to 192.168.202.40: rc = -22 [ 1512.427076] LNet: Skipped 1 previous similar message [ 1527.094956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1527.256234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1527.289098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1527.340409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.40@tcp --health 500 [ 1527.454781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1534.343112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1534.403292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1534.440818] LNet: 67699:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1534.452724] LNetError: 67699:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1534.457552] LNetError: 67699:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1534.465261] LNet: Removed LNI 192.168.202.40@tcp [ 1534.476051] LNet: Removed LNI 192.168.202.40@tcp1 [ 1534.510481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1534.576097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1534.598659] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1535.335500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1535.353133] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 1536.256793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1536.295570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1536.357269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1536.385720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1536.424832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1536.456766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1536.486864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1536.515460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.40@tcp --health 0 [ 1537.631920] LNet: There was an unexpected network error while writing to 192.168.202.40: rc = -22 [ 1537.639533] LNet: Skipped 3 previous similar messages [ 1537.948588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1548.403339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1548.468299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1548.499044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1555.335023] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 08:37:17 (1782218237) [ 1555.828724] LNet: 68744:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1555.835687] LNetError: 68744:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1555.840989] LNetError: 68744:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1555.849477] LNet: Removed LNI 192.168.202.40@tcp [ 1555.857770] LNet: Removed LNI 192.168.202.40@tcp1 [ 1556.453185] Key type .llcrypt unregistered [ 1556.461528] Key type ._llcrypt unregistered [ 1563.803501] Key type ._llcrypt registered [ 1563.805631] Key type .llcrypt registered [ 1563.845732] LNet: 68967:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1563.845732] mask usage: [+|-] ... [ 1564.094094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1564.127377] alg: No test for adler32 (adler32-zlib) [ 1565.078455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1565.104457] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1574.675591] Lustre: DEBUG MARKER: Initial discovery [ 1574.774479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.140@tcp [ 1577.189565] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1577.433649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1577.464471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.140@tcp [ 1577.497498] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1577.523314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1577.551510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.140@tcp -r 1 -m GET -l 3 [ 1577.584825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.140@tcp -r 1 -m GET -e local_error [ 1577.653253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.140@tcp -r 1 -m PUT -l 6 [ 1578.465906] LNet: 1 local NIs in recovery (showing 1): 192.168.202.40@tcp [ 1579.378358] Lustre: DEBUG MARKER: Force oleg240-client.virtnet to discover 192.168.202.140@tcp (in background) [ 1579.575427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.140@tcp [ 1581.220253] Lustre: DEBUG MARKER: Force oleg240-server to discover 192.168.202.40@tcp [ 1582.688874] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1584.179748] Lustre: DEBUG MARKER: Wait for 69505 [ 1585.683956] Lustre: DEBUG MARKER: Finished wait on 69505 [ 1586.001962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1586.032753] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1586.069931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1586.389404] LNet: 69958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1586.397683] LNetError: 69958:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1587.439333] LNet: Removed LNI 192.168.202.40@tcp [ 1587.899505] Key type .llcrypt unregistered [ 1587.901482] Key type ._llcrypt unregistered [ 1607.042951] Key type ._llcrypt registered [ 1607.046292] Key type .llcrypt registered [ 1608.502270] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 08:38:09 (1782218289) [ 1608.923298] Key type .llcrypt unregistered [ 1608.926620] Key type ._llcrypt unregistered [ 1616.056123] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1616.368952] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1616.389698] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1616.417641] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1619.050981] Key type ._llcrypt registered [ 1619.052847] Key type .llcrypt registered [ 1619.098092] LNet: 71002:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1619.098092] mask usage: [+|-] ... [ 1619.320800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1619.342930] alg: No test for adler32 (adler32-zlib) [ 1620.385232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1620.476290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1620.496804] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1621.280278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1621.308288] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1629.712963] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 08:38:31 (1782218311) [ 1629.813243] LNet: 72028:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1629.831506] LNetError: 72028:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1629.841799] LNet: Removed LNI 10.1.2.3@tcp [ 1629.862064] LNet: Removed LNI 192.168.202.40@tcp [ 1630.681430] Key type .llcrypt unregistered [ 1630.685572] Key type ._llcrypt unregistered [ 1638.288593] Key type ._llcrypt registered [ 1638.292297] Key type .llcrypt registered [ 1638.334435] LNet: 72286:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1638.334435] mask usage: [+|-] ... [ 1638.538355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1638.569028] alg: No test for adler32 (adler32-zlib) [ 1639.575914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1639.643189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1639.679233] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1640.524504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1640.547261] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1641.933528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.40@tcp [ 1642.977479] LNetError: Unexpected error -22 connecting to 192.168.202.40@tcp at host 192.168.202.40:988 [ 1648.835867] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 08:38:50 (1782218330) [ 1649.184381] LNet: 73169:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1649.192971] LNetError: 73169:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1650.207169] LNet: Removed LNI 10.1.2.3@tcp [ 1650.211244] LNetError: 73169:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1650.231789] LNet: Removed LNI 192.168.202.40@tcp [ 1650.971547] Key type .llcrypt unregistered [ 1650.975532] Key type ._llcrypt unregistered [ 1658.749230] Key type ._llcrypt registered [ 1658.752592] Key type .llcrypt registered [ 1658.800621] LNet: 73400:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1658.800621] mask usage: [+|-] ... [ 1659.040806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1659.071744] alg: No test for adler32 (adler32-zlib) [ 1660.155447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1660.270843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1660.318460] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 1661.057731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1661.088143] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 1662.508550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.40@tcp1 --nid 192.168.202.40@tcp2 [ 1669.041172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.40@tcp1 192.168.202.40@tcp2 [ 1669.070707] LNetError: 74005:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.40@tcp1 to 192.168.202.40@tcp2 - Route table may be misconfigured [ 1669.105382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.40@tcp2 192.168.202.40@tcp1 [ 1677.168353] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 08:39:18 (1782218358) [ 1677.417472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1677.446119] LNet: 74576:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1677.457315] LNetError: 74576:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1677.467375] LNet: Removed LNI 192.168.202.40@tcp1 [ 1678.505819] LNet: Removed LNI 192.168.202.40@tcp2 [ 1678.528702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1678.630437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1678.670019] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1679.551717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1679.572267] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 1680.713226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 1680.769753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.40@tcp 192.168.202.40@tcp [ 1680.812283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.40@tcp1 192.168.202.40@tcp1 [ 1680.875927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.40@tcp -d 192.168.202.40@tcp -e network_timeout [ 1680.915118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -e network_timeout [ 1680.958118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -e network_timeout [ 1681.017267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -e network_timeout [ 1681.078148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 1681.097552] LNet: There was an unexpected network error while writing to 192.168.202.40: rc = -22 [ 1681.118503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1681.242281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1681.364573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1681.631123] LNet: 1 local NIs in recovery (showing 1): 192.168.202.40@tcp [ 1688.356683] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 08:39:29 (1782218369) [ 1688.439837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1688.467904] LNet: 75438:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1688.473735] LNetError: 75438:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1688.484384] LNetError: 75438:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1688.487850] LNet: Removed LNI 192.168.202.40@tcp [ 1689.538530] LNet: Removed LNI 192.168.202.40@tcp1 [ 1689.589107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1689.694584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1690.036348] LNet: 75479:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1690.524723] Key type .llcrypt unregistered [ 1690.526830] Key type ._llcrypt unregistered [ 1703.605095] Key type ._llcrypt registered [ 1703.607740] Key type .llcrypt registered [ 1705.211537] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 08:39:46 (1782218386) [ 1705.466445] LNet: 76249:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1705.466445] mask usage: [+|-] ... [ 1705.680477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1705.699855] alg: No test for adler32 (adler32-zlib) [ 1706.710707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1708.281294] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1710.410986] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 08:39:51 (1782218391) [ 1710.485674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1710.509796] LNet: 76574:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1710.527216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1710.614145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1710.654123] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1711.478364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1711.505729] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 1712.325419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 1712.367471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp1 [ 1712.402762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp1 [ 1718.916862] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 08:40:00 (1782218400) [ 1720.911874] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg240-server [ 1722.824900] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 08:40:04 (1782218404) [ 1724.601648] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg240-server [ 1726.655726] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 08:40:08 (1782218408) [ 1729.410863] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg240-server [ 1731.486101] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 08:40:12 (1782218412) [ 1733.425862] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg240-server [ 1736.532366] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 08:40:17 (1782218417) [ 1738.141350] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg240-server [ 1739.728853] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 08:40:21 (1782218421) [ 1742.777985] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg240-server [ 1745.501938] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 08:40:26 (1782218426) [ 1748.532773] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg240-server [ 1750.532406] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 08:40:31 (1782218431) [ 1751.938721] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg240-server [ 1755.011980] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 08:40:35 (1782218435) [ 1756.999635] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg240-server [ 1758.613925] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 08:40:40 (1782218440) [ 1758.952577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1758.972964] LNet: 80459:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1758.994389] LNetError: 80459:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1759.003586] LNet: Removed LNI 192.168.202.40@tcp [ 1760.039844] LNet: Removed LNI 192.168.202.40@tcp1 [ 1760.076774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1760.158515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1760.199431] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1760.844736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1760.923913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 1761.097882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1761.115361] LNet: 80606:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1761.120746] LNetError: 80606:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1761.128317] LNetError: 80606:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1762.149263] LNet: Removed LNI 192.168.202.40@tcp [ 1762.211986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1762.338959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1762.386216] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1763.154476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 1764.556392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1764.572269] LNet: 80761:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1764.575201] LNetError: 80761:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1765.604720] LNet: Removed LNI 192.168.202.40@tcp [ 1765.632537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1765.688847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1765.710862] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1766.374928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1766.452753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 1766.603712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1766.622149] LNetError: 80908:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1766.644850] LNet: Removed LNI 192.168.202.40@tcp [ 1766.678932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1766.750750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1766.792591] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1767.733738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 1767.984693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1769.065838] LNet: Removed LNI 192.168.202.40@tcp [ 1769.107575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1769.176128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1769.216469] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1770.119538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1770.186635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 1770.413256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1770.452348] LNet: 81200:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1770.461982] LNet: 81200:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1771.557648] LNet: Removed LNI 192.168.202.40@tcp [ 1771.583529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1771.694826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1771.718441] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1772.432430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 1772.645299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1772.677728] LNetError: 81346:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1772.681067] LNetError: 81346:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1772.716256] LNet: Removed LNI 192.168.202.40@tcp [ 1772.736281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1772.806103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1772.834249] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1773.606580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1773.688851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 1773.914899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1774.954261] LNet: Removed LNI 192.168.202.40@tcp [ 1774.992751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1775.072786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1775.118043] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1775.933912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 1776.141507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1777.192915] LNet: Removed LNI 192.168.202.40@tcp [ 1777.248330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1777.335550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1777.369846] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1777.998334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1778.068467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 1778.201476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1779.239342] LNet: Removed LNI 192.168.202.40@tcp [ 1779.274731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1779.357412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1779.394271] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1780.113308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 1780.339649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1780.371348] LNet: 81931:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1780.378186] LNet: 81931:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1781.413732] LNet: Removed LNI 192.168.202.40@tcp [ 1781.480930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1781.530295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1781.547072] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1782.227375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1782.288282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 1782.508291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1782.537661] LNetError: 82079:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1782.542922] LNetError: 82079:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1782.559648] LNet: Removed LNI 192.168.202.40@tcp [ 1782.583891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1782.659785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1782.689622] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1783.461993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 1783.778963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1784.875067] LNet: Removed LNI 192.168.202.40@tcp [ 1784.924463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1785.004198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1785.035675] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1785.669521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1785.728897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1785.762488] LNet: Removed LNI 192.168.202.40@tcp [ 1785.780826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1785.836666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1785.854315] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1786.641159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1793.698521] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 08:41:15 (1782218475) [ 1793.946884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1793.997810] LNet: Removed LNI 192.168.202.40@tcp [ 1794.027553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1794.080202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1794.115158] LNet: Added LNI 192.168.202.40@tcp231 [8/256/0/180] [ 1795.383858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1795.424590] LNet: Removed LNI 192.168.202.40@tcp231 [ 1795.454566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1795.543789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1795.573782] LNet: Added LNI 192.168.202.40@tcp231 [8/256/0/180] [ 1795.578951] LNet: 83207:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1796.102823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1796.163360] LNet: Removed LNI 192.168.202.40@tcp231 [ 1796.203854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1796.277569] LNet: Added LNI 192.168.202.40@tcp231 [8/256/0/180] [ 1797.981937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1797.997900] LNet: 83352:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1798.002692] LNet: 83352:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1798.021541] LNet: Removed LNI 192.168.202.40@tcp231 [ 1798.038886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1798.133420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1798.151031] LNet: Added LNI 192.168.202.40@tcp231 [8/256/0/0] [ 1798.158388] LNet: 83360:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1798.774316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1798.820169] LNet: Removed LNI 192.168.202.40@tcp231 [ 1798.850413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1798.878120] LNet: Added LNI 192.168.202.40@tcp231 [8/256/0/0] [ 1800.166959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1800.203709] LNetError: 83503:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1800.208097] LNetError: 83503:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1800.233282] LNet: Removed LNI 192.168.202.40@tcp231 [ 1800.249957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1800.316684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1800.331033] LNet: Added LNI 192.168.202.40@tcp231 [8/256/0/60] [ 1800.335291] LNet: 83511:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1801.073794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1801.106704] LNet: Removed LNI 192.168.202.40@tcp231 [ 1801.120379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1801.175902] LNet: Added LNI 192.168.202.40@tcp231 [8/256/0/60] [ 1802.706719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1802.746274] LNet: Removed LNI 192.168.202.40@tcp231 [ 1802.762845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1802.816431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1802.842812] LNet: Added LNI 192.168.202.40@tcp231 [8/256/0/180] [ 1802.853453] LNet: 83663:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1803.411753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1803.441161] LNet: Removed LNI 192.168.202.40@tcp231 [ 1803.461926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1803.499694] LNet: Added LNI 192.168.202.40@tcp231 [8/256/0/180] [ 1812.658160] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 08:41:34 (1782218494) [ 1812.744273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1812.798078] LNet: Removed LNI 192.168.202.40@tcp231 [ 1812.831236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1812.923652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1812.952990] LNet: Added LNI 192.168.202.40@tcp232 [8/256/0/180] [ 1813.549127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1813.575601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1813.600668] LNet: Removed LNI 192.168.202.40@tcp232 [ 1813.619918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1813.641735] LNet: Added LNI 192.168.202.40@tcp232 [8/256/0/180] [ 1814.324845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1814.363793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1814.399392] LNet: Removed LNI 192.168.202.40@tcp232 [ 1814.420372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1814.457339] LNet: Added LNI 192.168.202.40@tcp232 [8/256/0/180] [ 1815.141578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1815.189254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1815.211662] LNet: Removed LNI 192.168.202.40@tcp232 [ 1815.232893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1815.250541] LNet: Added LNI 192.168.202.40@tcp232 [8/256/0/180] [ 1815.798609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1815.828808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1815.851049] LNet: Removed LNI 192.168.202.40@tcp232 [ 1815.864607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1821.953284] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 08:41:43 (1782218503) [ 1823.420278] Key type .llcrypt unregistered [ 1823.422168] Key type ._llcrypt unregistered [ 1830.266524] Key type ._llcrypt registered [ 1830.293909] Key type .llcrypt registered [ 1830.368401] LNet: 85746:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1830.368401] mask usage: [+|-] ... [ 1830.816187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1830.861037] alg: No test for adler32 (adler32-zlib) [ 1831.808774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1831.854588] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1839.763381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.140@tcp [ 1849.263983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1849.312928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1849.341130] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 1850.573453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_interrupt -r 2 [ 1850.600536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e local_interrupt -r 2 [ 1850.626542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_interrupt -r 2 [ 1850.653376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e local_interrupt -r 2 [ 1850.677603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e local_interrupt -r 2 [ 1850.703395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e local_interrupt -r 2 [ 1850.732954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e local_interrupt -r 2 [ 1850.763442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e local_interrupt -r 2 [ 1850.796939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.40@tcp 192.168.202.140@tcp [ 1850.816673] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1850.821461] LNet: 1 local NIs in recovery (showing 1): 192.168.202.40@tcp [ 1851.090761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1851.128274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1851.165813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1851.196273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1851.223668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1851.256754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1851.299832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1851.344831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1851.374083] LNet: 86180:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1851.406774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1851.441752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1851.476846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1851.518704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1851.563716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1851.602178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1852.073171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_dropped -r 2 [ 1852.104164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e local_dropped -r 2 [ 1852.134379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_dropped -r 2 [ 1852.178668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e local_dropped -r 2 [ 1852.214667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e local_dropped -r 2 [ 1852.259719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e local_dropped -r 2 [ 1852.301199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e local_dropped -r 2 [ 1852.344377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e local_dropped -r 2 [ 1852.369872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.40@tcp 192.168.202.140@tcp [ 1852.385486] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1852.633939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1852.660673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1852.690450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1852.715264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1852.739941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1852.772944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1852.820522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1852.862516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1852.896727] LNet: 86301:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1852.953985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1852.998067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1853.064695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1853.131558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1853.218247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1853.290208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1853.968110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_aborted -r 2 [ 1853.999786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e local_aborted -r 2 [ 1854.030933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_aborted -r 2 [ 1854.074707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e local_aborted -r 2 [ 1854.110692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e local_aborted -r 2 [ 1854.150869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e local_aborted -r 2 [ 1854.194506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e local_aborted -r 2 [ 1854.255299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e local_aborted -r 2 [ 1854.304529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.40@tcp 192.168.202.140@tcp [ 1854.335961] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1854.613370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1854.648175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1854.684860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1854.725094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1854.751272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1854.784757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1854.813326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1854.845811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1854.863340] LNet: 86421:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1854.882225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1854.913859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1854.947810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1854.977736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1855.009776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1855.045535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1855.719450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_no_route -r 2 [ 1855.754768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e local_no_route -r 2 [ 1855.796848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_no_route -r 2 [ 1855.829866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e local_no_route -r 2 [ 1855.875749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e local_no_route -r 2 [ 1855.948808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e local_no_route -r 2 [ 1855.995335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e local_no_route -r 2 [ 1856.031763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e local_no_route -r 2 [ 1856.093913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.40@tcp 192.168.202.140@tcp [ 1856.417395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1856.443449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1856.472135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1856.506880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1856.539362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1856.578866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1856.620726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1856.656410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1856.692140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1856.723435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1856.759622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1856.792958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1856.822436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1856.851456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1857.362988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e local_timeout -r 2 [ 1857.396930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e local_timeout -r 2 [ 1857.427908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e local_timeout -r 2 [ 1857.461936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e local_timeout -r 2 [ 1857.517470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e local_timeout -r 2 [ 1857.580800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e local_timeout -r 2 [ 1857.612657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e local_timeout -r 2 [ 1857.651954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e local_timeout -r 2 [ 1857.694771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.40@tcp 192.168.202.140@tcp [ 1857.720650] LNet: There was an unexpected network error while writing to 192.168.202.140: rc = -22 [ 1857.723495] LNet: Skipped 1 previous similar message [ 1858.013563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1858.048793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1858.097906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1858.143996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1858.184632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1858.208937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1858.261819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1858.315054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1858.336995] LNet: 86662:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1858.346560] LNet: 86662:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1858.389696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1858.433320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1858.475856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1858.507589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1858.566802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1858.605422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1859.188472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp -m GET -e remote_dropped -r 2 [ 1859.251484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp -d 192.168.202.140@tcp1 -m GET -e remote_dropped -r 2 [ 1859.300515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp -m GET -e remote_dropped -r 2 [ 1859.363689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp -d 192.168.202.40@tcp1 -m GET -e remote_dropped -r 2 [ 1859.422158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp -m GET -e remote_dropped -r 2 [ 1859.458752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.40@tcp1 -d 192.168.202.140@tcp1 -m GET -e remote_dropped -r 2 [ 1859.507661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp -m GET -e remote_dropped -r 2 [ 1859.551779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.40@tcp1 -d 192.168.202.40@tcp1 -m GET -e remote_dropped -r 2 [ 1859.605188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.40@tcp 192.168.202.140@tcp [ 1859.626864] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.140@tcp [ 1859.838256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1859.875354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1859.903753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1859.927611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1859.957986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1859.982931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1860.016460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1860.049897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1860.080800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1860.117044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1860.150889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1860.184836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1860.214365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1860.239589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1866.177079] LNet: 86856:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1866.192686] LNetError: 86856:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1866.201195] LNet: Removed LNI 192.168.202.40@tcp [ 1866.242680] LNet: Removed LNI 192.168.202.40@tcp1 [ 1866.765434] Key type .llcrypt unregistered [ 1866.768705] Key type ._llcrypt unregistered [ 1885.219897] Key type ._llcrypt registered [ 1885.223333] Key type .llcrypt registered [ 1886.361738] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 08:42:48 (1782218568) [ 1886.627732] LNet: 87664:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1886.627732] mask usage: [+|-] ... [ 1886.777562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1886.792627] alg: No test for adler32 (adler32-zlib) [ 1887.662484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1887.697143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1887.714992] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1888.271817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1888.297644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.40@tcp [ 1888.312880] LNetError: 87837:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.202.40@tcp to recovery [ 1888.415847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.40@tcp [ 1888.526550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1894.339584] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 08:42:56 (1782218576) [ 1894.533518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1894.555185] LNet: 88426:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1894.559906] LNetError: 88426:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1894.574542] LNet: Removed LNI 192.168.202.40@tcp [ 1894.588639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1894.637641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1894.661944] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1895.316171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1895.336723] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 1896.300769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.40@tcp [ 1896.315900] LNetError: 88692:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.202.40@tcp to recovery [ 1896.671361] LNet: 1 local NIs in recovery (showing 1): 192.168.202.40@tcp [ 1896.673059] LNet: There was an unexpected network error while writing to 192.168.202.40: rc = -22 [ 1897.533601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.40@tcp [ 1905.858478] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 08:43:07 (1782218587) [ 1907.350029] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg240-server [ 1909.473236] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 08:43:10 (1782218590) [ 1911.104887] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg240-server [ 1913.018303] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 08:43:14 (1782218594) [ 1915.001993] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg240-server [ 1916.759433] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 08:43:18 (1782218598) [ 1916.870750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1916.895215] LNet: 90317:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1916.910182] LNetError: 90317:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1916.918212] LNet: Removed LNI 192.168.202.40@tcp [ 1916.934065] LNet: Removed LNI 192.168.202.40@tcp1 [ 1917.028180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1917.114965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1917.173789] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1917.997536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1918.087815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1918.148435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1918.253662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1918.323643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1918.386770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1925.990588] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 08:43:27 (1782218607) [ 1926.117339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1926.140221] LNet: 91045:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1926.163558] LNetError: 91045:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1926.171385] LNetError: 91045:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1926.191378] LNet: Removed LNI 192.168.202.40@tcp [ 1926.233826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1926.314797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1926.338513] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1933.748601] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 08:43:35 (1782218615) [ 1933.913220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1933.936348] LNet: 91748:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1933.952911] LNetError: 91748:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1933.977284] LNet: Removed LNI 192.168.202.40@tcp [ 1934.054967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1934.108513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1934.129593] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1942.156266] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 08:43:43 (1782218623) [ 1943.679848] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1945.161544] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 08:43:47 (1782218627) [ 1945.654489] LNet: 92788:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1945.659692] LNetError: 92788:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1945.684054] LNet: Removed LNI 192.168.202.40@tcp [ 1946.322310] Key type .llcrypt unregistered [ 1946.333643] Key type ._llcrypt unregistered [ 1952.634387] Key type ._llcrypt registered [ 1952.636542] Key type .llcrypt registered [ 1952.667583] LNet: 93017:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1952.667583] mask usage: [+|-] ... [ 1952.821512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1952.834587] alg: No test for adler32 (adler32-zlib) [ 1953.764752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1953.793625] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 1961.682882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.140@tcp [ 1967.147596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.140@tcp [ 1967.506987] LNet: 93206:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1967.520355] LNetError: 93206:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1967.541863] LNet: Removed LNI 192.168.202.40@tcp [ 1968.159349] Key type .llcrypt unregistered [ 1968.160595] Key type ._llcrypt unregistered [ 1980.331670] Key type ._llcrypt registered [ 1980.336421] Key type .llcrypt registered [ 1981.737402] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1983.431317] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1985.090285] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 08:44:26 (1782218666) [ 1986.760538] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1988.653859] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 08:44:30 (1782218670) [ 1990.537451] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg240-server [ 1992.056725] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 08:44:33 (1782218673) [ 1993.904496] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg240-server [ 1995.767284] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 08:44:37 (1782218677) [ 1996.797991] LNet: 95258:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1996.797991] mask usage: [+|-] ... [ 1997.176873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1997.292072] alg: No test for adler32 (adler32-zlib) [ 1998.513347] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 1998.839537] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 2000.495188] Key type lgssc registered [ 2002.097359] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2023.210594] Key type lgssc unregistered [ 2023.517548] LNet: 96380:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2023.524296] LNetError: 96380:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2023.532920] LNet: Removed LNI 192.168.202.40@tcp [ 2024.352567] Key type .llcrypt unregistered [ 2024.353910] Key type ._llcrypt unregistered [ 2042.597796] Key type ._llcrypt registered [ 2042.600151] Key type .llcrypt registered [ 2043.964922] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 08:45:25 (1782218725) [ 2044.335773] LNet: 97177:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2044.335773] mask usage: [+|-] ... [ 2044.445245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2044.458433] alg: No test for adler32 (adler32-zlib) [ 2045.434970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2045.503059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2045.561515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2045.609422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2045.647717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2051.049399] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 08:45:32 (1782218732) [ 2051.247615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2051.308199] LNet: 97791:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2051.355585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2051.455805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2051.501454] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 2052.332172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2052.349186] LNet: 97939:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2052.352900] LNetError: 97939:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2052.378876] LNet: Removed LNI 192.168.202.40@tcp [ 2052.398467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2052.456796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2052.486185] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 2053.002784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2053.026541] LNetError: 97967:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2053.039179] LNet: Removed LNI 192.168.202.40@tcp [ 2053.063634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2053.113802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2053.135655] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 2060.854501] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 08:45:42 (1782218742) [ 2061.018471] LNet: 98558:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2061.027545] LNet: 98558:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2061.035717] LNetError: 98558:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2061.043838] LNet: Removed LNI 192.168.202.40@tcp [ 2061.602500] Key type .llcrypt unregistered [ 2061.604598] Key type ._llcrypt unregistered [ 2067.387700] Key type ._llcrypt registered [ 2067.389827] Key type .llcrypt registered [ 2067.462869] LNet: 98791:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2067.462869] mask usage: [+|-] ... [ 2067.658249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2067.681506] alg: No test for adler32 (adler32-zlib) [ 2068.639995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2068.791621] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 2068.799744] LNet: *** cfs_fail_loc=e003, val=100*** [ 2069.542397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 2069.562438] LNet: *** cfs_fail_loc=e003, val=100*** [ 2069.564657] LNet: Skipped 2 previous similar messages [ 2069.571954] Ping reply received with mapping: [ 2069.573724] found metadata reply NI[192.168.202.40@tcp] [ 2069.575847] 28 ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2069.578430] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2069.580577] 5a 5a 5a 5a 5a 5a 5a 5a [ 2069.740516] LNet: 98957:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2069.755873] LNetError: 98957:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2069.779632] LNet: Removed LNI 192.168.202.40@tcp [ 2070.468326] Key type .llcrypt unregistered [ 2070.469849] Key type ._llcrypt unregistered [ 2082.419385] Key type ._llcrypt registered [ 2082.421515] Key type .llcrypt registered [ 2083.779946] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 08:46:05 (1782218765) [ 2084.430705] Key type .llcrypt unregistered [ 2084.437347] Key type ._llcrypt unregistered [ 2091.122942] Key type ._llcrypt registered [ 2091.124531] Key type .llcrypt registered [ 2091.176745] LNet: 99952:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2091.176745] mask usage: [+|-] ... [ 2091.409028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2091.458049] alg: No test for adler32 (adler32-zlib) [ 2092.664476] LNetError: 100002:0:(api-ni.c:2779:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2093.059608] Key type .llcrypt unregistered [ 2093.064952] Key type ._llcrypt unregistered [ 2098.140567] Key type ._llcrypt registered [ 2098.145265] Key type .llcrypt registered [ 2099.304393] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 08:46:21 (1782218781) [ 2099.810700] Key type .llcrypt unregistered [ 2099.812688] Key type ._llcrypt unregistered [ 2105.503586] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2105.507219] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2107.975558] Key type ._llcrypt registered [ 2107.977423] Key type .llcrypt registered [ 2108.003427] LNet: 100870:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2108.003427] mask usage: [+|-] ... [ 2108.128119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2108.136434] alg: No test for adler32 (adler32-zlib) [ 2109.044955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2109.090965] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2117.480363] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 08:46:39 (1782218799) [ 2117.595528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2117.632645] LNet: 101782:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2117.639701] LNetError: 101782:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2117.651405] LNet: Removed LNI 10.1.2.3@tcp [ 2117.664557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2118.157393] Key type .llcrypt unregistered [ 2118.159798] Key type ._llcrypt unregistered [ 2124.485935] Key type ._llcrypt registered [ 2124.487386] Key type .llcrypt registered [ 2124.552195] LNet: 102014:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2124.552195] mask usage: [+|-] ... [ 2124.722452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2124.742067] alg: No test for adler32 (adler32-zlib) [ 2125.753339] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 2125.818613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 2125.967906] LNet: 102071:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2125.972972] LNetError: 102071:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2127.026923] LNet: Removed LNI 192.168.202.40@tcp [ 2127.647477] Key type .llcrypt unregistered [ 2127.649080] Key type ._llcrypt unregistered [ 2138.855360] Key type ._llcrypt registered [ 2138.856983] Key type .llcrypt registered [ 2140.033954] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 08:47:01 (1782218821) [ 2140.702264] Key type .llcrypt unregistered [ 2140.704021] Key type ._llcrypt unregistered [ 2147.483665] Key type ._llcrypt registered [ 2147.485411] Key type .llcrypt registered [ 2147.521767] LNet: 103055:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2147.521767] mask usage: [+|-] ... [ 2147.756794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2147.788071] alg: No test for adler32 (adler32-zlib) [ 2154.498229] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 08:47:16 (1782218836) [ 2154.798469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2154.853849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2154.927833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2154.945249] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 2155.940847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2155.991470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2156.044165] LNet: Removed LNI 192.168.202.40@tcp [ 2156.369687] LNet: 103814:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2156.879901] Key type .llcrypt unregistered [ 2156.881657] Key type ._llcrypt unregistered [ 2168.146337] Key type ._llcrypt registered [ 2168.148727] Key type .llcrypt registered [ 2169.437651] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 08:47:31 (1782218851) [ 2169.750321] LNet: 104579:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2169.750321] mask usage: [+|-] ... [ 2169.877091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2169.898961] alg: No test for adler32 (adler32-zlib) [ 2170.856887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2170.893604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2170.923672] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 2171.542727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 2171.576711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.40@tcp [ 2171.593515] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2171.596626] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2171.598898] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2171, 0, 0 [ 2171.601896] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2171.604017] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2171.606734] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2171.608974] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2171, 0, 0 [ 2171.612644] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2171.614633] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2171.617333] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2171.619740] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2171, 0, 2221 [ 2171.623645] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2171.625834] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2171.628630] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2171.631619] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2171, 0, 2221 [ 2171.635257] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2178.004418] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 08:47:39 (1782218859) [ 2178.534215] LNet: 105353:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2178.537737] LNetError: 105353:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2178.558466] LNet: Removed LNI 192.168.202.40@tcp [ 2179.342307] Key type .llcrypt unregistered [ 2179.344965] Key type ._llcrypt unregistered [ 2185.998186] Key type ._llcrypt registered [ 2185.999919] Key type .llcrypt registered [ 2186.047827] LNet: 105577:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2186.047827] mask usage: [+|-] ... [ 2186.300996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2186.322324] alg: No test for adler32 (adler32-zlib) [ 2187.294865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2187.318332] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 2195.132198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.140@tcp [ 2205.123659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2205.157178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2205.186263] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 2206.495984] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2206.639737] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2206.647921] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2206.660740] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2209.097048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2209.118797] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2219.258804] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 08:48:21 (1782218901) [ 2219.337958] LNet: 106854:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2219.349362] LNetError: 106854:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2220.383463] LNet: Removed LNI 192.168.202.40@tcp [ 2220.391157] LNetError: 106854:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2220.401735] LNet: Removed LNI 192.168.202.40@tcp1 [ 2220.424978] LNet: Removed LNI 10.1.2.3@tcp99 [ 2220.960481] Key type .llcrypt unregistered [ 2220.962623] Key type ._llcrypt unregistered [ 2227.136624] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2227.148289] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2229.644655] Key type ._llcrypt registered [ 2229.648308] Key type .llcrypt registered [ 2229.698376] LNet: 107127:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2229.698376] mask usage: [+|-] ... [ 2229.927351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2229.943756] alg: No test for adler32 (adler32-zlib) [ 2230.898202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2230.958554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2231.000800] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 2232.038317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2232.058315] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2234.200727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2234.263157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2234.331416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2234.357938] LNetError: 107606:0:(peer.c:2234:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2234.399389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2234.452623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 2235.029372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2235.145271] LNet: 107626:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2235.156550] LNetError: 107626:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2235.171685] LNet: Removed LNI 10.1.2.3@tcp [ 2236.202566] LNet: Removed LNI 192.168.202.40@tcp [ 2237.051580] Key type .llcrypt unregistered [ 2237.053873] Key type ._llcrypt unregistered [ 2248.743735] Key type ._llcrypt registered [ 2248.744930] Key type .llcrypt registered [ 2250.342210] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 08:48:51 (1782218931) [ 2250.582439] LNet: 108420:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2250.582439] mask usage: [+|-] ... [ 2250.798443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2250.820850] alg: No test for adler32 (adler32-zlib) [ 2251.823969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2251.879553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2251.902231] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 2252.611508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.40@tcp [ 2252.698069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg240-client@tcp [ 2258.624893] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 08:49:00 (1782218940) [ 2258.819054] LNet: 109157:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2258.823158] LNetError: 109157:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2258.853980] LNet: Removed LNI 192.168.202.40@tcp [ 2259.448500] Key type .llcrypt unregistered [ 2259.450252] Key type ._llcrypt unregistered [ 2266.217564] Key type ._llcrypt registered [ 2266.225100] Key type .llcrypt registered [ 2266.546865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2266.590062] alg: No test for adler32 (adler32-zlib) [ 2267.539755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2267.561059] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 2267.594874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2275.162986] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 08:49:16 (1782218956) [ 2277.062630] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2279.015476] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 08:49:20 (1782218960) [ 2280.759600] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2282.366714] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 08:49:24 (1782218964) [ 2282.460730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2282.496943] LNet: 110574:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2282.504802] LNetError: 110574:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2282.532335] LNet: Removed LNI 192.168.202.40@tcp [ 2282.583896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2282.768286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2282.803042] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 2283.669814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2283.703989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2289.998753] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 08:49:31 (1782218971) [ 2290.050910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2290.060648] LNet: 111286:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2290.068115] LNetError: 111286:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2290.088939] LNet: Removed LNI 192.168.202.40@tcp [ 2290.136757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2290.209518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2290.243969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2290.783596] Key type .llcrypt unregistered [ 2290.787773] Key type ._llcrypt unregistered [ 2302.874529] Key type ._llcrypt registered [ 2302.876642] Key type .llcrypt registered [ 2304.397786] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 08:49:45 (1782218985) [ 2304.772336] LNet: 112099:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2304.772336] mask usage: [+|-] ... [ 2304.910952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2304.924660] alg: No test for adler32 (adler32-zlib) [ 2305.886634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2305.958378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2305.984721] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 2306.602498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2306.664974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 2313.682242] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 08:49:55 (1782218995) [ 2313.770445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2313.807961] LNet: 112842:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2313.816980] LNetError: 112842:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2313.836343] LNet: Removed LNI 192.168.202.40@tcp [ 2313.868501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2313.901288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2313.939889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2321.020669] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 08:50:02 (1782219002) [ 2321.179390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2321.207152] LNet: 113421:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2321.240131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2321.342994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2321.571565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2321.604880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2321.647820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2322.101334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2322.138633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2322.209922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2322.225492] LNet: 113460:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2322.244659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2322.312387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2322.893058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2322.935108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2323.008482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2323.549252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2323.577337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2330.615323] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 08:50:12 (1782219012) [ 2330.734208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2330.748067] LNet: 114054:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2330.780954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2330.857825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2330.878475] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 2331.357341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2331.366703] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 2332.336876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2332.377968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 2332.551353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2332.575659] LNetError: 114340:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2332.581295] LNet: Removed LNI 192.168.202.40@tcp [ 2332.596738] LNet: Removed LNI 192.168.202.40@tcp2 [ 2332.619414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2332.671924] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 2332.676325] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 2332.693574] LNet: 114346:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2333.109393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 2333.360197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2334.431363] LNet: Removed LNI 192.168.202.40@tcp [ 2334.433911] LNetError: 114383:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2334.441766] LNetError: 114383:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2334.465063] LNet: Removed LNI 192.168.202.40@tcp2 [ 2334.500993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2334.609185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2334.646155] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 2335.442060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2335.481385] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 2336.546302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 2336.607381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2336.656541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 2336.784804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2336.800940] LNet: 114668:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2336.803711] LNet: 114668:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2336.807694] LNetError: 114668:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2336.811450] LNet: Removed LNI 192.168.202.40@tcp [ 2337.836874] LNet: Removed LNI 192.168.202.40@tcp2 [ 2337.928346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2338.031726] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 2338.036736] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 2338.042150] LNet: 114674:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2338.648772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 2339.034085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2339.076327] LNet: Removed LNI 192.168.202.40@tcp [ 2340.135476] LNet: Removed LNI 192.168.202.40@tcp2 [ 2340.204939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2340.304843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2340.336816] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 2341.312446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2341.333358] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 2342.312540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2342.395951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 2342.600328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2342.618438] LNetError: 114990:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2342.622415] LNetError: 114990:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2343.649480] LNet: Removed LNI 192.168.202.40@tcp [ 2343.662044] LNet: Removed LNI 192.168.202.40@tcp2 [ 2343.721896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2343.871666] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 2343.891676] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 2343.893824] LNet: 114997:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2344.892367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 2345.202213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2345.242572] LNet: 115034:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2345.249404] LNet: 115034:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2346.271545] LNet: Removed LNI 192.168.202.40@tcp [ 2346.282363] LNet: Removed LNI 192.168.202.40@tcp2 [ 2346.315744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2346.433621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2346.463430] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 2347.276665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2347.295524] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 2348.774663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 2348.847504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2348.960747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 2349.380901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2350.431232] LNet: Removed LNI 192.168.202.40@tcp [ 2351.463056] LNet: Removed LNI 192.168.202.40@tcp2 [ 2351.505785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2351.649890] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 2351.686681] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 2351.698185] LNet: 115325:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2352.670463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 2352.841631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2352.859170] LNetError: 115362:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2352.862951] LNetError: 115362:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2352.867721] LNet: Removed LNI 192.168.202.40@tcp [ 2353.905649] LNet: Removed LNI 192.168.202.40@tcp2 [ 2353.993874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2354.081141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2354.111157] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 2355.281074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2355.317092] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 2356.601142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2356.646761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2356.751250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 2357.346203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2358.432211] LNet: Removed LNI 192.168.202.40@tcp [ 2358.452112] LNet: Removed LNI 192.168.202.40@tcp2 [ 2358.532784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2358.679880] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 2358.689856] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 2358.699538] LNet: 115665:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2359.659846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 2360.042286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2361.119819] LNet: Removed LNI 192.168.202.40@tcp [ 2361.135682] LNet: Removed LNI 192.168.202.40@tcp2 [ 2361.185842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2361.291168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2361.340717] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 2362.565159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2362.609933] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 2364.022629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 2364.072659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2364.125312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2364.220311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 2364.785505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2364.846421] LNet: 116019:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2364.859262] LNet: 116019:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2365.919230] LNet: Removed LNI 192.168.202.40@tcp [ 2366.952940] LNet: Removed LNI 192.168.202.40@tcp2 [ 2366.986950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2367.088847] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 2367.097666] LNet: Added LNI 192.168.202.40@tcp2 [8/256/0/180] [ 2367.102539] LNet: 116025:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2367.824462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 2375.240572] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2376.791853] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 08:50:58 (1782219058) [ 2377.037336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2377.062767] LNetError: 116784:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2377.067337] LNetError: 116784:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2377.071979] LNet: Removed LNI 192.168.202.40@tcp [ 2378.153956] LNet: Removed LNI 192.168.202.40@tcp2 [ 2378.192511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2378.531316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2378.550905] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 2379.192675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2379.212249] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2380.192657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2380.227627] LNet: Removed LNI 10.1.2.3@tcp [ 2380.249578] LNet: Removed LNI 192.168.202.40@tcp [ 2380.277710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2380.333510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2380.353928] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 2380.361816] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2380.855979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2380.916030] LNet: Removed LNI 192.168.202.40@tcp [ 2380.937173] LNet: Removed LNI 10.1.2.3@tcp [ 2388.410142] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 08:51:09 (1782219069) [ 2388.726213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2388.752831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2388.827148] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2388.991256] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2388.994861] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2388.998558] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2391.407682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2391.444484] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 2392.269845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2392.289747] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2394.487053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.40@tcp [ 2394.545979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2394.631922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.40@tcp -m PUT -e local_timeout -r 1 [ 2394.731667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2394.789352] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2394.793125] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2395.043201] LNetError: 117719:0:(lib-move.c:4714:lnet_parse()) 192.168.202.40@tcp, src 192.168.202.40@tcp: Dropping PUT (error -108 looking up sender) [ 2396.066489] LNet: Removed LNI 10.1.2.3@tcp [ 2397.108833] LNet: Removed LNI 192.168.202.40@tcp [ 2397.632398] Key type .llcrypt unregistered [ 2397.634264] Key type ._llcrypt unregistered [ 2410.870496] Key type ._llcrypt registered [ 2410.872787] Key type .llcrypt registered [ 2412.118613] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 08:51:33 (1782219093) [ 2412.498558] LNet: 118949:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2412.498558] mask usage: [+|-] ... [ 2412.770329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2412.800379] alg: No test for adler32 (adler32-zlib) [ 2413.896605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2414.569754] LNet: 119030:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2415.316337] Key type .llcrypt unregistered [ 2415.317969] Key type ._llcrypt unregistered [ 2422.195486] Key type ._llcrypt registered [ 2422.197804] Key type .llcrypt registered [ 2422.232888] LNet: 119259:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2422.232888] mask usage: [+|-] ... [ 2422.416952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2422.437894] alg: No test for adler32 (adler32-zlib) [ 2423.440543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2423.493881] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 2431.334344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.140@tcp [ 2431.568943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2432.162772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.140@tcp [ 2432.348935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2432.368668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2432.393648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2432.440629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2432.462405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2432.495372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2432.517686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2432.547170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2432.562580] LNet: 119480:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2432.576633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2432.598658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2432.621796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2432.650504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2432.677861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2432.719838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2433.239295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2433.369039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2433.387550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2433.401802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2433.415290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2433.435914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2433.451141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2433.465839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2433.481730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2433.490761] LNet: 119597:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2433.501366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2433.523488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2433.549578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2433.573411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2433.596682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2433.618122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2434.163611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.140@tcp [ 2434.435748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2434.468401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2434.496697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2434.537312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2434.573901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2434.599334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2434.626248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2434.665224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2434.689695] LNet: 119714:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2434.706215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2434.723551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2434.753335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2434.776491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2434.783433] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.140@tcp [ 2434.803709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2434.837485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2435.362791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2435.635865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2435.653404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2435.672762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2435.696570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2435.723379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2435.757193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2435.800386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2435.835595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2435.874428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2435.905291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2435.937476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2435.968870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2436.029754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2436.082475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2436.392665] LNet: 119877:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2436.399170] LNetError: 119877:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2437.414607] LNet: Removed LNI 192.168.202.40@tcp [ 2437.901756] Key type .llcrypt unregistered [ 2437.904364] Key type ._llcrypt unregistered [ 2458.279242] Key type ._llcrypt registered [ 2458.280851] Key type .llcrypt registered [ 2459.981251] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 08:52:21 (1782219141) [ 2460.237891] LNet: 120683:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2460.237891] mask usage: [+|-] ... [ 2460.449160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2460.465211] alg: No test for adler32 (adler32-zlib) [ 2461.407368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2461.820763] LNet: 120765:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2462.513278] Key type .llcrypt unregistered [ 2462.523727] Key type ._llcrypt unregistered [ 2469.272253] Key type ._llcrypt registered [ 2469.278993] Key type .llcrypt registered [ 2469.369513] LNet: 120994:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2469.369513] mask usage: [+|-] ... [ 2469.701371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2469.712558] alg: No test for adler32 (adler32-zlib) [ 2470.708507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2470.737587] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 2478.402825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.140@tcp [ 2488.003426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2488.036995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2488.056804] LNet: Added LNI 192.168.202.40@tcp1 [8/256/0/180] [ 2489.147445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2489.759610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.140@tcp [ 2490.030304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2490.086579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2490.127488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2490.185666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2490.207351] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.140@tcp [ 2490.223780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2490.260229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2490.325024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2490.388527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2490.413981] LNet: 121417:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2490.445371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2490.488159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2490.539769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2490.606380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2490.669991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2490.743967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2491.502957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2491.773350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2491.827654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2491.857369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2491.904784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2491.938760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2491.964660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2492.001771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2492.039203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2492.052907] LNet: 121534:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2492.079896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2492.113869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2492.142292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2492.168928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2492.216544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2492.260292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2492.890975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.140@tcp [ 2493.129394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2493.167897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2493.200298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2493.237629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2493.271295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2493.294517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2493.320330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2493.346739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2493.359955] LNet: 121652:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2493.380877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2493.406322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2493.456938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2493.508693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2493.547691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2493.588839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2494.282044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2494.588341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2494.618669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2494.650350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2494.681863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2494.708161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2494.738795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2494.818675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2494.881537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2494.962748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2495.006597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2495.052829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2495.111897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2495.144940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2495.179318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2501.234772] LNet: 121850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2501.247780] LNetError: 121850:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2501.262343] LNet: Removed LNI 192.168.202.40@tcp [ 2501.283328] LNet: Removed LNI 192.168.202.40@tcp1 [ 2501.952353] Key type .llcrypt unregistered [ 2501.956295] Key type ._llcrypt unregistered [ 2521.328742] Key type ._llcrypt registered [ 2521.334437] Key type .llcrypt registered [ 2522.795892] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 08:53:24 (1782219204) [ 2530.986803] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 08:53:32 (1782219212) [ 2538.063602] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 08:53:39 (1782219219) [ 2545.322973] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 08:53:47 (1782219227) [ 2552.065240] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 08:53:53 (1782219233) [ 2559.582344] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 08:54:01 (1782219241) [ 2566.012822] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 08:54:07 (1782219247) [ 2572.506654] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 08:54:14 (1782219254) [ 2579.917057] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 08:54:21 (1782219261) [ 2587.556848] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 08:54:29 (1782219269) [ 2594.514503] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 08:54:36 (1782219276) [ 2601.618549] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 08:54:43 (1782219283) [ 2608.863379] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 08:54:50 (1782219290) [ 2616.971076] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 08:54:58 (1782219298) [ 2624.580530] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 08:55:06 (1782219306) [ 2633.311514] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 08:55:14 (1782219314) [ 2642.943395] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 08:55:23 (1782219323) [ 2650.293540] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 08:55:32 (1782219332) [ 2650.551566] LNet: 132390:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2650.551566] mask usage: [+|-] ... [ 2650.727322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2650.747689] alg: No test for adler32 (adler32-zlib) [ 2651.626335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2651.793802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2651.834702] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 2653.467461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2653.506116] LNet: 132561:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2653.523436] LNetError: 132561:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2653.535827] LNet: Removed LNI 192.168.202.40@tcp [ 2653.551435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2653.609027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2653.637426] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 2662.055915] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 08:55:43 (1782219343) [ 2669.679468] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 08:55:51 (1782219351) [ 2677.938981] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 08:55:59 (1782219359) [ 2685.361288] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 08:56:07 (1782219367) [ 2692.876800] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 08:56:14 (1782219374) [ 2700.066995] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 08:56:22 (1782219382) [ 2707.312665] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 08:56:29 (1782219389) [ 2716.873478] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 08:56:38 (1782219398) [ 2725.706841] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 08:56:47 (1782219407) [ 2733.083789] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 08:56:54 (1782219414) [ 2739.879083] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 08:57:01 (1782219421) [ 2745.987602] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 08:57:07 (1782219427) [ 2753.644801] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2466 sec ========== 08:57:15 (1782219435) [ 2754.038810] LNet: 140027:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2754.044793] LNetError: 140027:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2754.066406] LNet: Removed LNI 192.168.202.40@tcp [ 2754.888469] Key type .llcrypt unregistered [ 2754.891441] Key type ._llcrypt unregistered [ 2770.271172] Key type ._llcrypt registered [ 2770.281134] Key type .llcrypt registered [ 2770.517283] LNet: 140504:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2770.517283] mask usage: [+|-] ... [ 2770.777560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2770.793454] alg: No test for adler32 (adler32-zlib) [ 2771.904252] Lustre: Lustre: Build Version: 2.17.54_83_gf3505cf [ 2772.312300] LNet: Added LNI 192.168.202.40@tcp [8/256/0/180] [ 2774.031729] Key type lgssc registered [ 2775.505619] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2833.674787] Lustre: Mounted lustre-client [ 2833.944397] LNet: 141972:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2833.944397] mask usage: [+|-] ... [ 2839.407174] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2847.734190] Lustre: Unmounted lustre-client [ 2885.762255] Key type lgssc unregistered [ 2886.023236] LNet: 142956:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2886.044856] LNetError: 142956:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2887.116955] LNet: Removed LNI 192.168.202.40@tcp [ 2888.149158] Key type .llcrypt unregistered [ 2888.150852] Key type ._llcrypt unregistered