[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 353555248 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002342] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.006824] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007016] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008008] pid_max: default: 32768 minimum: 301 [ 0.010068] LSM: Security Framework initializing [ 0.011034] Yama: becoming mindful. [ 0.012030] SELinux: Initializing. [ 0.013047] *** VALIDATE selinux *** [ 0.020738] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024426] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025101] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026064] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028008] *** VALIDATE tmpfs *** [ 0.028815] *** VALIDATE proc *** [ 0.029153] *** VALIDATE cgroup *** [ 0.030005] *** VALIDATE cgroup2 *** [ 0.031177] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032170] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034026] Spectre V2 : User space: Vulnerable [ 0.035003] Speculative Store Bypass: Vulnerable [ 0.037906] debug: unmapping init [mem 0xffffffff93859000-0xffffffff93860fff] [ 0.039757] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040799] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041016] ... version: 2 [ 0.042006] ... bit width: 48 [ 0.042939] ... generic registers: 4 [ 0.043010] ... value mask: 0000ffffffffffff [ 0.044011] ... max period: 00007fffffffffff [ 0.045024] ... fixed-purpose events: 3 [ 0.046008] ... event mask: 000000070000000f [ 0.048270] rcu: Hierarchical SRCU implementation. [ 0.050723] smp: Bringing up secondary CPUs ... [ 0.051583] x86: Booting SMP configuration: [ 0.052019] .... node #0, CPUs: #1 #2 #3 [ 0.056077] smp: Brought up 1 node, 4 CPUs [ 0.057957] smpboot: Max logical packages: 1 [ 0.058010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.126703] node 0 deferred pages initialised in 66ms [ 0.129196] devtmpfs: initialized [ 0.130258] x86/mm: Memory block size: 128MB [ 0.133036] gcov: version magic: 0x41383552 [ 0.135304] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.136082] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.137311] pinctrl core: initialized pinctrl subsystem [ 0.138250] [ 0.138834] ************************************************************* [ 0.139011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.140010] ** ** [ 0.141014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.142008] ** ** [ 0.143010] ** This means that this kernel is built to expose internal ** [ 0.144007] ** IOMMU data structures, which may compromise security on ** [ 0.145007] ** your system. ** [ 0.146011] ** ** [ 0.147011] ** If you see this message and you are not debugging the ** [ 0.148009] ** kernel, report this immediately to your vendor! ** [ 0.149010] ** ** [ 0.150008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.151011] ************************************************************* [ 0.152772] NET: Registered protocol family 16 [ 0.153387] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.154054] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.155036] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.156431] cpuidle: using governor menu [ 0.157578] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.159424] PCI: Using configuration type 1 for base access [ 0.160117] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.168140] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.169039] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.171065] cryptd: max_cpu_qlen set to 1000 [ 0.172220] ACPI: Added _OSI(Module Device) [ 0.173010] ACPI: Added _OSI(Processor Device) [ 0.173798] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.175007] ACPI: Added _OSI(Processor Aggregator Device) [ 0.178423] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.182355] ACPI: Interpreter enabled [ 0.183039] ACPI: PM: (supports S0 S3 S4 S5) [ 0.184008] ACPI: Using IOAPIC for interrupt routing [ 0.185019] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.187356] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.195854] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.197020] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.198007] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.200055] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.203053] acpiphp: Slot [2] registered [ 0.203975] acpiphp: Slot [5] registered [ 0.205085] acpiphp: Slot [6] registered [ 0.205981] acpiphp: Slot [3] registered [ 0.207051] acpiphp: Slot [4] registered [ 0.208072] acpiphp: Slot [7] registered [ 0.208978] acpiphp: Slot [8] registered [ 0.210043] acpiphp: Slot [9] registered [ 0.210964] acpiphp: Slot [10] registered [ 0.211043] acpiphp: Slot [11] registered [ 0.211941] acpiphp: Slot [12] registered [ 0.213061] acpiphp: Slot [13] registered [ 0.214066] acpiphp: Slot [14] registered [ 0.215045] acpiphp: Slot [15] registered [ 0.216087] acpiphp: Slot [16] registered [ 0.218077] acpiphp: Slot [17] registered [ 0.219093] acpiphp: Slot [18] registered [ 0.221066] acpiphp: Slot [19] registered [ 0.222073] acpiphp: Slot [20] registered [ 0.223050] acpiphp: Slot [21] registered [ 0.223846] acpiphp: Slot [22] registered [ 0.225045] acpiphp: Slot [23] registered [ 0.225952] acpiphp: Slot [24] registered [ 0.227056] acpiphp: Slot [25] registered [ 0.227793] acpiphp: Slot [26] registered [ 0.229046] acpiphp: Slot [27] registered [ 0.229837] acpiphp: Slot [28] registered [ 0.230040] acpiphp: Slot [29] registered [ 0.231059] acpiphp: Slot [30] registered [ 0.232054] acpiphp: Slot [31] registered [ 0.232856] PCI host bridge to bus 0000:00 [ 0.234011] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.235007] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.237010] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.238009] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.240009] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.241013] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.243176] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.245691] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.247928] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.254018] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.258054] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.260017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.262009] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.263010] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.265464] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.267733] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.270041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.272662] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.275886] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.284013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.287009] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.290606] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.296014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.300013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.311014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.317989] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.327015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.331012] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.348014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.355998] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.359360] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.361353] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.362279] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.364142] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.369738] iommu: Default domain type: Passthrough [ 0.370633] SCSI subsystem initialized [ 0.371000] ACPI: bus type USB registered [ 0.374149] usbcore: registered new interface driver usbfs [ 0.377070] usbcore: registered new interface driver hub [ 0.379107] usbcore: registered new device driver usb [ 0.381317] pps_core: LinuxPPS API ver. 1 registered [ 0.382005] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.385065] PTP clock support registered [ 0.387038] EDAC MC: Ver: 3.0.0 [ 0.388108] PCI: Using ACPI for IRQ routing [ 0.389768] NetLabel: Initializing [ 0.391009] NetLabel: domain hash size = 128 [ 0.392006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.394082] NetLabel: unlabeled traffic allowed by default [ 0.397031] vgaarb: loaded [ 0.398247] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.400017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.407451] clocksource: Switched to clocksource kvm-clock [ 0.495929] VFS: Disk quotas dquot_6.6.0 [ 0.497032] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.498622] *** VALIDATE ramfs *** [ 0.499569] *** VALIDATE hugetlbfs *** [ 0.500677] pnp: PnP ACPI init [ 0.502305] pnp: PnP ACPI: found 6 devices [ 0.515701] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.518095] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.520029] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.521496] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.523445] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.524897] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.526897] NET: Registered protocol family 2 [ 0.529316] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.532876] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.535187] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.539524] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.541767] TCP: Hash tables configured (established 65536 bind 65536) [ 0.544118] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.546883] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.548991] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.551025] NET: Registered protocol family 1 [ 0.552670] RPC: Registered named UNIX socket transport module. [ 0.554217] RPC: Registered udp transport module. [ 0.555566] RPC: Registered tcp transport module. [ 0.557098] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.559118] NET: Registered protocol family 44 [ 0.560325] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.561845] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.563345] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.565222] PCI: CLS 0 bytes, default 64 [ 0.566395] Unpacking initramfs... [ 1.989387] debug: unmapping init [mem 0xffff9882bcc64000-0xffff9882bffcffff] [ 1.993868] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.995795] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.997553] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.583922] Initialise system trusted keyrings [ 2.585481] Key type blacklist registered [ 2.586985] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.595539] zbud: loaded [ 2.598478] *** VALIDATE nfs *** [ 2.599516] *** VALIDATE nfs4 *** [ 2.601041] pstore: using deflate compression [ 2.607564] Platform Keyring initialized [ 2.776654] NET: Registered protocol family 38 [ 2.788056] Key type asymmetric registered [ 2.790087] Asymmetric key parser 'x509' registered [ 2.792656] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.796371] io scheduler mq-deadline registered [ 2.798047] io scheduler kyber registered [ 2.800173] io scheduler bfq registered [ 2.808562] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.815268] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.819481] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.823398] ACPI: Power Button [PWRF] [ 2.829619] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.835350] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.843718] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.877055] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.907177] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.925872] Non-volatile memory driver v1.3 [ 2.927309] Linux agpgart interface v0.103 [ 2.968116] virtio_blk virtio1: [vda] 146656 512-byte logical blocks (75.1 MB/71.6 MiB) [ 2.970783] vda: detected capacity change from 0 to 75087872 [ 2.986331] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.989672] vdb: detected capacity change from 0 to 1073741824 [ 2.997718] libphy: Fixed MDIO Bus: probed [ 3.012962] usbcore: registered new interface driver usbserial_generic [ 3.015962] usbserial: USB Serial support registered for generic [ 3.020432] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.025532] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.027859] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.030521] mousedev: PS/2 mouse device common for all mice [ 3.034062] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.037862] rtc_cmos 00:05: RTC can wake from S4 [ 3.040409] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.046182] rtc_cmos 00:05: registered as rtc0 [ 3.049335] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.050766] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.058648] intel_pstate: CPU model not supported [ 3.061977] hid: raw HID events driver (C) Jiri Kosina [ 3.063980] usbcore: registered new interface driver usbhid [ 3.065850] usbhid: USB HID core driver [ 3.067313] drop_monitor: Initializing network drop monitor service [ 3.069432] Initializing XFRM netlink socket [ 3.071353] NET: Registered protocol family 10 [ 3.074204] Segment Routing with IPv6 [ 3.075673] NET: Registered protocol family 17 [ 3.077908] mpls_gso: MPLS GSO support [ 3.084857] RAS: Correctable Errors collector initialized. [ 3.086736] AVX version of gcm_enc/dec engaged. [ 3.088131] AES CTR mode by8 optimization enabled [ 3.167364] sched_clock: Marking stable (3167174220, 0)->(3884652812, -717478592) [ 3.172890] registered taskstats version 1 [ 3.174728] Loading compiled-in X.509 certificates [ 3.176792] zswap: loaded using pool lzo/zbud [ 3.260845] Key type big_key registered [ 3.280772] Key type encrypted registered [ 3.283099] ima: No TPM chip found, activating TPM-bypass! [ 3.285639] ima: Allocated hash algorithm: sha1 [ 3.288426] ima: No architecture policies found [ 3.290806] evm: Initialising EVM extended attributes: [ 3.293675] evm: security.selinux [ 3.294932] evm: security.ima [ 3.295983] evm: security.capability [ 3.297484] evm: HMAC attrs: 0x1 [ 3.299757] rtc_cmos 00:05: setting system clock to 2026-09-13 01:39:18 UTC (1789263558) [ 3.306448] debug: unmapping init [mem 0xffffffff94803000-0xffffffff949fffff] [ 3.328260] debug: unmapping init [mem 0xffffffff93582000-0xffffffff93858fff] [ 3.341274] Write protecting the kernel read-only data: 28672k [ 3.344715] debug: unmapping init [mem 0xffffffff91c03000-0xffffffff91dfffff] [ 3.347392] debug: unmapping init [mem 0xffffffff92514000-0xffffffff925fffff] [ 3.395980] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.404215] systemd[1]: Detected virtualization kvm. [ 3.405649] systemd[1]: Detected architecture x86-64. [ 3.407260] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.439419] systemd[1]: No hostname configured. [ 3.441659] systemd[1]: Set hostname to . [ 3.444407] random: systemd: uninitialized urandom read (16 bytes read) [ 3.447598] systemd[1]: Initializing machine ID from random generator. [ 3.617496] random: systemd: uninitialized urandom read (16 bytes read) [ 3.620293] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.632361] random: systemd: uninitialized urandom read (16 bytes read) [ 3.635100] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.642906] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Journal Service. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.457449] device-mapper: uevent: version 1.0.3 [ 4.459350] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook...[ 5.331716] random: fast init done [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.508403] virtio_net virtio0 ens2: renamed from eth0 [ 5.917191] scsi host0: ata_piix [ 6.244414] scsi host1: ata_piix [ 6.246950] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 6.251404] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 10.805267] random: crng init done [ 10.809143] random: 7 urandom warning(s) missed due to ratelimiting [ 12.117767] dracut-initqueue[587]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 13.598590] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.601894] printk: systemd: 23 output lines suppressed due to ratelimiting [ 16.008306] SELinux: Disabled at runtime. [ 16.083922] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.097667] systemd[1]: Detected virtualization kvm. [ 16.100611] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 17.012161] systemd[1]: initrd-switch-root.service: Succeeded. [ 17.015715] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 17.027466] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 17.033185] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 17.036274] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 17.043242] systemd[1]: Starting Journal Service... Starting Journal Service... [ 17.050390] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ 17.265851] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 18.276553] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 18.804280] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 18.864393] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 19.184569] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.216552] EDAC sbridge: Ver: 1.1.2 [ 21.465642] Key type dns_resolver registered [ 21.953727] NFS: Registering the id_resolver key type [ 21.956370] Key type id_resolver registered [ 21.958262] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg252-client login: [ 73.509758] libcfs: loading out-of-tree module taints kernel. [ 73.626307] Key type ._llcrypt registered [ 73.659975] Key type .llcrypt registered [ 73.905657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 73.914707] alg: No test for adler32 (adler32-zlib) [ 75.009075] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 75.464746] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 77.152180] Key type lgssc registered [ 78.141541] Lustre: Echo OBD driver; http://www.lustre.org/ [ 135.138538] Lustre: Mounted lustre-client - version 2.17.57_103_g3c4c45f [ 137.860550] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 145.461937] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing check_logdir /tmp/testlogs/ [ 147.191483] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing yml_node [ 149.095479] Lustre: DEBUG MARKER: Client: 2.17.57.103 [ 150.147799] Lustre: DEBUG MARKER: MDS: 2.17.57.103 [ 151.297168] Lustre: DEBUG MARKER: OSS: 2.17.57.103 [ 152.034224] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sat Sep 12 21:41:46 EDT 2026 [ 157.856315] Lustre: Unmounted lustre-client [ 187.636360] Key type lgssc unregistered [ 187.828298] LNet: 8399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 187.836226] LNetError: 8399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 187.852263] LNet: Removed LNI 192.168.202.52@tcp [ 188.299126] Key type .llcrypt unregistered [ 188.301594] Key type ._llcrypt unregistered [ 200.877090] Key type ._llcrypt registered [ 200.878213] Key type .llcrypt registered [ 200.952208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 200.961709] alg: No test for adler32 (adler32-zlib) [ 201.821562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 201.856095] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 201.868597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 203.976887] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 204.020397] LNet: 9267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 204.024735] LNetError: 9267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 204.038745] LNet: Removed LNI 192.168.202.52@tcp [ 204.392108] Key type .llcrypt unregistered [ 204.394303] Key type ._llcrypt unregistered [ 206.967747] Key type ._llcrypt registered [ 206.969026] Key type .llcrypt registered [ 207.598812] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 21:42:42 (1789263762) [ 207.784878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 207.792555] alg: No test for adler32 (adler32-zlib) [ 208.611850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 208.651483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 208.664923] LNet: 9683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 208.675763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 208.703164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 208.712253] LNet: 9690:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 211.824509] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 21:42:46 (1789263766) [ 211.870936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 211.881919] LNet: 10261:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 211.894327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 211.933043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 215.104791] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 21:42:49 (1789263769) [ 215.131414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 215.141033] LNet: 10844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 215.152377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 215.200505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 218.405482] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 21:42:52 (1789263772) [ 218.490657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 218.508154] LNet: 11433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 218.531205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 218.576366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 221.920462] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 21:42:56 (1789263776) [ 221.971656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 222.000506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 222.048883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 222.272873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 225.649300] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 21:43:00 (1789263780) [ 225.720802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 225.738215] LNet: 12607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 225.751622] LNet: 12607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 225.769788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 225.812019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 226.039416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 226.285930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 230.269869] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 21:43:04 (1789263784) [ 230.340220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 230.359594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 230.416762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 234.454446] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 21:43:08 (1789263788) [ 234.569734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 234.580211] LNet: 13787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 234.583692] LNet: 13787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 234.601988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 234.636813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 234.655225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 234.901518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 234.931053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 235.154701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 235.173279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 235.425557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 235.452726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 235.660347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 235.682175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 235.907734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 235.933649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 236.202659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 236.231429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 236.449430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 236.473175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 236.695768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 236.717626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 236.947728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 236.963565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 237.189230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 237.213926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 237.436413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 237.467470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 237.686501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 237.711114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 241.359412] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 21:43:15 (1789263795) [ 241.460494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 241.492607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 241.534224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 241.556768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 245.015156] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 21:43:19 (1789263799) [ 245.093200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 245.127202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 245.203989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 245.231336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 248.964918] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 21:43:23 (1789263803) [ 249.023392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 249.044718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 249.113618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 249.143259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 252.834766] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 21:43:27 (1789263807) [ 252.902133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 252.927601] LNet: 16220:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 252.930162] LNet: 16220:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 252.945861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 253.008388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 253.033235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 256.953715] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 21:43:31 (1789263811) [ 257.025954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 257.058809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 257.119078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 257.145857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 260.865717] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 21:43:35 (1789263815) [ 260.898220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 260.925551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 260.976310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 260.994391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 265.115919] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 21:43:39 (1789263819) [ 265.163242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 265.222814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 265.288668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 265.569655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 269.288948] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 21:43:43 (1789263823) [ 269.363592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 269.405199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 269.491447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 269.630386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 273.186501] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 21:43:47 (1789263827) [ 273.269301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 273.292643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 273.386650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 273.529066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 277.133437] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 21:43:51 (1789263831) [ 277.224740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 277.269445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 277.524762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 277.678886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 281.683491] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 21:43:56 (1789263836) [ 281.790548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 281.813691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 281.865555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 281.900393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 285.749387] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 21:44:00 (1789263840) [ 285.837070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 285.857772] LNet: 21051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 285.860837] LNet: 21051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 285.872737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 285.934806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 285.980424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 289.721257] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 21:44:04 (1789263844) [ 289.803970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 289.839604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 289.865896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 293.472687] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 21:44:07 (1789263847) [ 293.560979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 293.584848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 293.623304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 297.275533] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 21:44:11 (1789263851) [ 297.367427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 297.394451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.435454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 297.645762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 301.291239] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 21:44:15 (1789263855) [ 301.377235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 301.411827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 301.462722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 301.493271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 305.236899] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 21:44:19 (1789263859) [ 305.385878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 305.411544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 305.487355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 305.512760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 309.299462] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 21:44:23 (1789263863) [ 309.364249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 309.401848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 309.466950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 309.488315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 313.104522] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 21:44:27 (1789263867) [ 313.146314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 313.185123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 313.229592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 313.249297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 316.768313] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 21:44:31 (1789263871) [ 316.841828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 316.890961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 316.934663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 316.977991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 316.989778] LNetError: 25774:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 317.013233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 317.033688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 320.089159] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 21:44:34 (1789263874) [ 320.202775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 320.225839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 320.259534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.sqmwfdth1D [ 320.308159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.sqmwfdth1D [ 320.338339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.sqmwfdth1D [ 323.643832] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 21:44:38 (1789263878) [ 323.715628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 323.749081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 323.779577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 323.799417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 323.833256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 327.100947] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 21:44:41 (1789263881) [ 327.212339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 327.236187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 327.285932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 327.308025] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 328.051894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.52@tcp [ 328.709758] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 330.440188] Key type lgssc registered [ 331.002085] Lustre: Echo OBD driver; http://www.lustre.org/ [ 332.941415] LNetError: 27534:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.202.52@tcp99 has no available nets [ 334.530441] Key type lgssc unregistered [ 334.753222] LNetError: 28639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 334.765112] LNet: Removed LNI 192.168.202.52@tcp [ 335.138465] Key type .llcrypt unregistered [ 335.140176] Key type ._llcrypt unregistered [ 338.162329] Key type ._llcrypt registered [ 338.164722] Key type .llcrypt registered [ 338.913830] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 21:44:53 (1789263893) [ 339.143090] Key type .llcrypt unregistered [ 339.146979] Key type ._llcrypt unregistered [ 342.500262] Key type ._llcrypt registered [ 342.501602] Key type .llcrypt registered [ 342.603307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 342.617340] alg: No test for adler32 (adler32-zlib) [ 343.466393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 343.515899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 343.531516] LNet: 29550:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 343.537251] LNet: 29550:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 343.542388] LNet: 29550:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 343.552379] LNet: Message forwarding will be enabled in 2s [ 346.656143] LNet: Message forwarding enabled [ 346.835385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 346.845602] LNet: Message forwarding will stop in 2s to 6s [ 350.754372] LNet: Message forwarding disabled [ 351.140748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 351.152199] LNet: 29649:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 351.156150] LNet: 29649:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 351.159919] LNet: 29649:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 351.167657] LNet: Message forwarding will be enabled in 2s [ 351.252387] LNet: 29662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 351.563533] Key type .llcrypt unregistered [ 351.565364] Key type ._llcrypt unregistered [ 356.884496] Key type ._llcrypt registered [ 356.886803] Key type .llcrypt registered [ 357.746832] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 21:45:11 (1789263911) [ 358.186719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 358.197270] alg: No test for adler32 (adler32-zlib) [ 359.093926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.136416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 364.498488] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 21:45:18 (1789263918) [ 364.708613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 364.770699] LNet: 31055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 364.827361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 364.974359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 365.020474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 365.061087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 365.091378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 365.119748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 365.147892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 365.198655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 365.232592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 365.260139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 365.285733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 365.333827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 365.363926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 365.408743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 365.463250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 365.528574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 365.582993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 365.637555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 365.687784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 365.729872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 365.765806] LNetError: 31105:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 369.697807] hrtimer: interrupt took 3687782 ns [ 373.239925] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 21:45:27 (1789263927) [ 373.338239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 373.380766] LNet: 31668:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 373.417221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 373.601768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 381.494450] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 21:45:35 (1789263935) [ 383.610553] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg252-server [ 386.181270] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 21:45:39 (1789263939) [ 386.434511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 386.473267] LNet: 32607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 386.501677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 386.566025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 386.605765] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 387.944375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[53-57]@tcp [ 387.962211] LNet: 32741:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 388.057593] LNetError: 32611:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.53@tcp is being used as a gateway but routing feature is not turned on [ 388.083400] LNetError: 32611:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.53@tcp has gone from up to down [ 388.102751] LNetError: 32611:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 391.074648] LNetError: 32615:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.55@tcp has gone from up to down [ 391.101656] LNetError: 32615:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 396.025484] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 21:45:50 (1789263950) [ 398.018930] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg252-server [ 400.357808] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 21:45:54 (1789263954) [ 400.729477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.776732] LNet: 33679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 400.783057] LNetError: 33679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 400.802374] LNet: Removed LNI 192.168.202.52@tcp [ 400.857252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.921358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 400.947335] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 401.727563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[53-58/2]@tcp [ 401.744156] LNet: 33796:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 401.749106] LNet: 33796:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 401.763496] LNetError: 33688:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.57@tcp has gone from up to down [ 401.769470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[53-58/2]@tcp [ 408.179935] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 21:46:02 (1789263962) [ 408.271164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.302821] LNet: 34385:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 408.319657] LNetError: 34385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 408.345120] LNet: Removed LNI 192.168.202.52@tcp [ 408.373403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.437727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 408.488668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.529262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.580644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 408.616791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.652772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.698617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 408.766215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.798639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.856772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 408.958669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.990743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 409.040550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 409.105621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 409.135475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 409.189239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 409.250825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 409.329652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 409.365310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 409.413700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 409.467786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 409.509323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 409.578508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 409.601108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 409.645516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 409.730408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 409.851221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 409.937159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 410.048125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 410.089259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 410.169323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 410.278396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 410.339695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 410.439292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 417.415342] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 21:46:11 (1789263971) [ 419.564508] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg252-server [ 421.285882] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 21:46:15 (1789263975) [ 423.029555] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg252-server [ 425.017834] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 21:46:18 (1789263978) [ 425.117540] LNet: 35782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 425.122711] LNet: 35782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 425.780519] Key type .llcrypt unregistered [ 425.786464] Key type ._llcrypt unregistered [ 432.250390] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 432.445910] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 432.448796] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 432.452185] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 434.946287] Key type ._llcrypt registered [ 434.948433] Key type .llcrypt registered [ 435.155791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 435.168697] alg: No test for adler32 (adler32-zlib) [ 436.081061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 436.175384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 436.217611] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 436.765886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 436.791536] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 438.741832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 438.757823] LNet: Removed LNI 10.1.2.3@tcp [ 438.868952] LNet: 36507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 438.871816] LNetError: 36507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 438.881476] LNet: Removed LNI 192.168.202.52@tcp [ 439.545561] Key type .llcrypt unregistered [ 439.549584] Key type ._llcrypt unregistered [ 450.629986] Key type ._llcrypt registered [ 450.631786] Key type .llcrypt registered [ 451.872394] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 21:46:45 (1789264005) [ 452.661348] Key type .llcrypt unregistered [ 452.662588] Key type ._llcrypt unregistered [ 458.233391] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 458.422123] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 458.427535] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 458.429673] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 459.847775] Key type ._llcrypt registered [ 459.852129] Key type .llcrypt registered [ 459.979489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 459.993893] alg: No test for adler32 (adler32-zlib) [ 460.992885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.030828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 461.047815] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 461.556556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 461.575338] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 463.533161] LNet: 37986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 463.547889] LNetError: 37986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 463.553698] LNet: Removed LNI 10.1.2.3@tcp [ 463.580441] LNet: Removed LNI 192.168.202.52@tcp [ 464.169270] Key type .llcrypt unregistered [ 464.171435] Key type ._llcrypt unregistered [ 474.148625] Key type ._llcrypt registered [ 474.151412] Key type .llcrypt registered [ 475.134536] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 21:47:09 (1789264029) [ 475.485329] Key type .llcrypt unregistered [ 475.487230] Key type ._llcrypt unregistered [ 479.772972] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 479.901780] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 479.910272] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 479.916966] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 482.348238] Key type ._llcrypt registered [ 482.350872] Key type .llcrypt registered [ 482.491655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 482.502662] alg: No test for adler32 (adler32-zlib) [ 483.376511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 483.418474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 483.437479] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 484.855715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 484.864750] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 486.276153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 486.302100] LNet: Removed LNI 10.1.2.3@tcp [ 486.313801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 486.344910] LNet: Removed LNI 10.1.2.31@tcp [ 486.360859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 486.378508] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 486.947362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 486.963886] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 487.537702] LNet: 39898:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 487.542980] LNetError: 39898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 487.548513] LNet: Removed LNI 10.1.2.31@tcp [ 487.555801] LNet: Removed LNI 10.1.2.3@tcp [ 487.910759] Key type .llcrypt unregistered [ 487.912038] Key type ._llcrypt unregistered [ 495.785850] Key type ._llcrypt registered [ 495.789877] Key type .llcrypt registered [ 496.742949] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 21:47:30 (1789264050) [ 497.020171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 497.032550] alg: No test for adler32 (adler32-zlib) [ 497.913280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 497.957367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 497.981627] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 498.549970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 498.565822] LNet: 40847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 498.569991] LNetError: 40847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 498.595458] LNet: Removed LNI 192.168.202.52@tcp [ 498.615947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 498.663549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.52@tcp [ 498.675470] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 499.204398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.52@tcp [ 499.244325] LNet: Removed LNI 192.168.202.52@tcp [ 499.312646] LNet: 40982:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 499.790442] Key type .llcrypt unregistered [ 499.793929] Key type ._llcrypt unregistered [ 508.344735] Key type ._llcrypt registered [ 508.346199] Key type .llcrypt registered [ 509.114155] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 21:47:43 (1789264063) [ 510.183904] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg252-server [ 511.289697] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 21:47:45 (1789264065) [ 511.894555] Key type .llcrypt unregistered [ 511.897140] Key type ._llcrypt unregistered [ 516.404345] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 516.522913] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 516.544096] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 516.560569] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 518.940658] Key type ._llcrypt registered [ 518.945243] Key type .llcrypt registered [ 519.098123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 519.105621] alg: No test for adler32 (adler32-zlib) [ 519.960240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.998342] LNet: Added LNI 192.168.202.52@tcp1 [8/256/0/180] [ 520.531647] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 522.058721] LNet: 42812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 522.062867] LNetError: 42812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 522.066360] LNet: Removed LNI 192.168.202.52@tcp1 [ 522.079810] LNet: Removed LNI 10.1.2.3@tcp2 [ 522.506387] Key type .llcrypt unregistered [ 522.508144] Key type ._llcrypt unregistered [ 529.947837] Key type ._llcrypt registered [ 529.950325] Key type .llcrypt registered [ 530.784434] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 21:48:05 (1789264085) [ 531.103919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 531.118450] alg: No test for adler32 (adler32-zlib) [ 531.989645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 532.035236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 532.048414] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 532.571529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 532.582434] LNet: 43778:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 532.585520] LNetError: 43778:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 532.598330] LNet: Removed LNI 192.168.202.52@tcp [ 532.608468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 532.649192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 532.664887] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 537.459900] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 21:48:11 (1789264091) [ 537.544536] LNet: 44362:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 537.548577] LNetError: 44362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 537.557038] LNet: Removed LNI 192.168.202.52@tcp [ 537.984662] Key type .llcrypt unregistered [ 537.986555] Key type ._llcrypt unregistered [ 541.848026] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 541.984773] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 541.989790] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 541.993820] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 544.356891] Key type ._llcrypt registered [ 544.359459] Key type .llcrypt registered [ 544.494839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 544.506419] alg: No test for adler32 (adler32-zlib) [ 545.358308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 545.389573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 545.403796] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 545.870900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 545.885721] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 547.170069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.184714] LNet: 45105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 547.192354] LNetError: 45105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 547.198424] LNet: Removed LNI 10.1.2.3@tcp [ 547.215059] LNet: Removed LNI 192.168.202.52@tcp [ 547.241678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.303204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 547.321711] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 547.325159] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 547.597062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.612261] LNet: Removed LNI 10.1.2.3@tcp [ 547.621821] LNet: Removed LNI 192.168.202.52@tcp [ 547.639831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.683211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 547.697853] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 547.700503] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 547.880411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 547.893051] LNet: Added LNI 192.168.202.52@tcp2 [8/256/0/180] [ 548.411882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 548.421381] LNet: 45294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 548.424312] LNet: 45294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 548.428766] LNetError: 45294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 548.431724] LNetError: 45294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 548.435062] LNet: Removed LNI 10.1.2.3@tcp [ 548.436840] LNet: Removed LNI 192.168.202.52@tcp [ 548.445349] LNet: Removed LNI 192.168.202.52@tcp2 [ 548.453876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 548.484813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 548.500224] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 548.504942] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 548.510286] LNet: Added LNI 192.168.202.52@tcp2 [8/256/0/180] [ 552.836633] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 21:48:27 (1789264107) [ 552.939431] LNet: 45888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 552.945362] LNetError: 45888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 552.953660] LNetError: 45888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 552.961609] LNet: Removed LNI 10.1.2.3@tcp [ 552.965129] LNet: Removed LNI 192.168.202.52@tcp [ 552.977741] LNet: Removed LNI 192.168.202.52@tcp2 [ 553.425529] Key type .llcrypt unregistered [ 553.428338] Key type ._llcrypt unregistered [ 557.387785] Key type ._llcrypt registered [ 557.390449] Key type .llcrypt registered [ 557.532710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 557.545939] alg: No test for adler32 (adler32-zlib) [ 558.443455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.485703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 558.500272] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 559.093107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 559.105222] LNet: 46282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 559.107964] LNetError: 46282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 559.116674] LNet: Removed LNI 192.168.202.52@tcp [ 559.129449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 559.169413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 559.186837] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 559.440273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 559.465431] LNet: Removed LNI 192.168.202.52@tcp [ 559.485159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 559.532204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 559.559703] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 559.833863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 559.850934] LNet: 46342:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 559.855847] LNet: 46342:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 559.860244] LNetError: 46342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 559.863285] LNetError: 46342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 559.882244] LNet: Removed LNI 192.168.202.52@tcp [ 559.899148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 559.946947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 559.965100] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 560.287260] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 560.458087] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 560.472234] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 560.475866] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 562.686807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 562.700600] LNet: 46421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 562.707766] LNetError: 46421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 562.731710] LNet: Removed LNI 192.168.202.52@tcp [ 562.754711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.810944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 562.833756] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 567.223976] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 21:48:41 (1789264121) [ 567.334553] LNet: 47014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 567.346828] LNetError: 47014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 567.367117] LNet: Removed LNI 192.168.202.52@tcp [ 567.897566] Key type .llcrypt unregistered [ 567.899199] Key type ._llcrypt unregistered [ 572.467922] Key type ._llcrypt registered [ 572.471653] Key type .llcrypt registered [ 572.705521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 572.719516] alg: No test for adler32 (adler32-zlib) [ 573.584424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 573.616616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 573.631962] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 574.264796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 574.276364] LNet: 47413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 574.280572] LNetError: 47413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 574.301765] LNet: Removed LNI 192.168.202.52@tcp [ 574.319686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 574.367823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 574.389921] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 579.068538] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 21:48:53 (1789264133) [ 579.193632] LNet: 48004:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 579.197656] LNetError: 48004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 579.220667] LNet: Removed LNI 192.168.202.52@tcp [ 579.668587] Key type .llcrypt unregistered [ 579.670721] Key type ._llcrypt unregistered [ 583.872372] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 584.003435] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 584.009619] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 584.014699] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 586.474496] Key type ._llcrypt registered [ 586.478313] Key type .llcrypt registered [ 586.617599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 586.636150] alg: No test for adler32 (adler32-zlib) [ 587.487580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 587.525608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 587.540921] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 588.064682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 588.091442] LNet: 48452:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 588.098176] LNetError: 48452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 588.113071] LNet: Removed LNI 192.168.202.52@tcp [ 588.139660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 588.205353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 588.238817] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 593.085187] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 21:49:07 (1789264147) [ 593.270371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 593.289692] LNet: 49050:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 593.294687] LNetError: 49050:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 593.304826] LNet: Removed LNI 192.168.202.52@tcp [ 593.320325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.392873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 598.130823] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 21:49:12 (1789264152) [ 598.279446] LNet: 49629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 598.822249] Key type .llcrypt unregistered [ 598.825378] Key type ._llcrypt unregistered [ 603.549357] Key type ._llcrypt registered [ 603.552125] Key type .llcrypt registered [ 603.690827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 603.707570] alg: No test for adler32 (adler32-zlib) [ 604.573120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.618851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 604.640394] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 605.289597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 605.308301] LNet: 50034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 605.313955] LNetError: 50034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 605.327935] LNet: Removed LNI 192.168.202.52@tcp [ 605.343454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.399263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 605.421190] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 610.471391] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 21:49:24 (1789264164) [ 610.628786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.658414] LNet: 50626:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 610.668158] LNetError: 50626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 610.683733] LNet: Removed LNI 192.168.202.52@tcp [ 610.702878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.754515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 615.235745] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 21:49:29 (1789264169) [ 616.282480] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 617.370414] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 21:49:31 (1789264171) [ 618.567377] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 619.839947] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 21:49:34 (1789264174) [ 620.855771] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 622.002711] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 21:49:36 (1789264176) [ 623.182719] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 624.650496] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 21:49:38 (1789264178) [ 625.951690] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 627.360436] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 21:49:41 (1789264181) [ 627.518783] LNet: 52674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 627.974136] Key type .llcrypt unregistered [ 627.977889] Key type ._llcrypt unregistered [ 632.000780] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 632.112025] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 632.116721] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 632.119975] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 634.547132] Key type ._llcrypt registered [ 634.550179] Key type .llcrypt registered [ 634.689218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 634.703628] alg: No test for adler32 (adler32-zlib) [ 635.542231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.594195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 635.619177] LNet: Added LNI 192.168.202.52@tcp [8/512/0/180] [ 635.623568] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 635.625672] LNet: 52994:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 635.694737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 635.708301] LNet: 53013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 635.712540] LNetError: 53013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 635.716638] LNet: Removed LNI 10.1.2.3@tcp [ 635.718674] LNetError: 53013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 635.732828] LNet: Removed LNI 192.168.202.52@tcp [ 635.748292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.796903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 635.814272] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 635.861933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 635.874693] LNet: 53037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 635.880746] LNetError: 53037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 635.894088] LNet: Removed LNI 192.168.202.52@tcp [ 635.917867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.967388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 635.982667] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 635.987039] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 635.989240] LNet: 53045:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 636.036235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 636.048820] LNet: 53063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 636.055313] LNetError: 53063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 636.057971] LNet: Removed LNI 10.1.2.3@tcp [ 636.059530] LNetError: 53063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 636.069730] LNet: Removed LNI 192.168.202.52@tcp [ 636.102396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 636.166363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 636.189139] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 636.192265] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 636.194620] LNet: 53071:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 636.297120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 636.315000] LNet: 53097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 636.319618] LNetError: 53097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 636.322958] LNet: Removed LNI 10.1.2.3@tcp [ 636.325611] LNetError: 53097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 636.337840] LNet: Removed LNI 192.168.202.52@tcp [ 636.355991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 636.408675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 636.428989] LNet: Added LNI 192.168.202.52@tcp [8/256/0/182] [ 636.432420] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 636.509017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 636.524945] LNet: 53125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 636.529442] LNetError: 53125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 636.532345] LNet: Removed LNI 10.1.2.3@tcp [ 636.534832] LNetError: 53125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 636.546768] LNet: Removed LNI 192.168.202.52@tcp [ 636.568435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 636.605433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 636.633934] LNet: Added LNI 192.168.202.52@tcp [8/256/0/182] [ 636.640941] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 636.643613] LNet: 53133:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 636.704496] LNet: Removed LNI 10.1.2.3@tcp [ 636.714993] LNet: Removed LNI 192.168.202.52@tcp [ 637.105727] Key type .llcrypt unregistered [ 637.109252] Key type ._llcrypt unregistered [ 644.314736] Key type ._llcrypt registered [ 644.315932] Key type .llcrypt registered [ 645.341754] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 21:49:59 (1789264199) [ 646.497433] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 647.573060] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 21:50:01 (1789264201) [ 647.845456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 647.856368] alg: No test for adler32 (adler32-zlib) [ 648.720303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.757989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 648.773570] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 649.305123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.52 [ 649.319910] LNet: 54394:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 649.329628] LNet: Removed LNI 192.168.202.52@tcp [ 653.837866] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 21:50:08 (1789264208) [ 653.919802] LNet: 54955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 653.926168] LNetError: 54955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 653.947579] LNet: Removed LNI 192.168.202.52@tcp [ 654.522579] Key type .llcrypt unregistered [ 654.525624] Key type ._llcrypt unregistered [ 658.421025] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 658.557255] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 661.060845] Key type ._llcrypt registered [ 661.063231] Key type .llcrypt registered [ 661.190542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 661.199722] alg: No test for adler32 (adler32-zlib) [ 662.363437] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 666.810871] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 21:50:21 (1789264221) [ 667.036616] LNet: 55874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 667.040767] LNetError: 55874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 667.053271] LNet: Removed LNI 10.1.2.3@tcp [ 667.523612] Key type .llcrypt unregistered [ 667.525936] Key type ._llcrypt unregistered [ 672.357361] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 672.555943] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 672.558630] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 672.562631] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 675.112174] Key type ._llcrypt registered [ 675.114293] Key type .llcrypt registered [ 675.211420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 675.226617] alg: No test for adler32 (adler32-zlib) [ 676.134648] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 680.716990] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 21:50:34 (1789264234) [ 681.827101] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 683.277759] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 21:50:37 (1789264237) [ 683.417957] LNet: 57054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 683.427165] LNetError: 57054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 683.443116] LNet: Removed LNI 10.1.2.3@tcp [ 684.126126] Key type .llcrypt unregistered [ 684.128665] Key type ._llcrypt unregistered [ 689.300979] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 689.589538] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 689.594971] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 689.598310] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 692.213306] Key type ._llcrypt registered [ 692.215533] Key type .llcrypt registered [ 692.389819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 692.403977] alg: No test for adler32 (adler32-zlib) [ 693.343323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 693.411400] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 699.731927] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 21:50:53 (1789264253) [ 700.111389] LNet: 58280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 700.116718] LNetError: 58280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 700.130125] LNet: Removed LNI 10.1.2.3@tcp [ 700.621354] Key type .llcrypt unregistered [ 700.624939] Key type ._llcrypt unregistered [ 705.039701] Key type ._llcrypt registered [ 705.041758] Key type .llcrypt registered [ 705.226324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 705.236818] alg: No test for adler32 (adler32-zlib) [ 706.071053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 706.091321] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 711.622242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.152@tcp [ 712.017927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp -m GET -e local_interrupt -r 1 [ 712.051252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp -m GET -e local_interrupt -r 1 [ 712.076241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.152@tcp [ 712.088636] LNet: There was an unexpected network error while writing to 192.168.202.152: rc = -22 [ 712.227328] LNet: 1 local NIs in recovery (showing 1): 192.168.202.52@tcp [ 712.264564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 712.281827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 712.305576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 712.329801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 712.354403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 712.374930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 712.398647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 712.421772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 712.437283] LNet: 58714:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 712.450214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 712.475110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 712.497766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 712.520906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 712.545948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 712.568891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 712.597462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 712.979829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp -m GET -e local_dropped -r 1 [ 712.997576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp -m GET -e local_dropped -r 1 [ 713.015958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.152@tcp [ 713.032261] LNet: There was an unexpected network error while writing to 192.168.202.152: rc = -22 [ 713.188328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 713.218911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 713.270527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 713.298077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 713.321629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 713.340988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 713.379503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 713.411057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 713.428605] LNet: 58824:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 713.442989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 713.466973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 713.501044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 713.539218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 713.574456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 713.604144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 713.637941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 714.006050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp -m GET -e local_aborted -r 1 [ 714.038436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp -m GET -e local_aborted -r 1 [ 714.063611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.152@tcp [ 714.081193] LNet: There was an unexpected network error while writing to 192.168.202.152: rc = -22 [ 714.086248] LNet: Skipped 1 previous similar message [ 714.254661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 714.276521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 714.299311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 714.317386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 714.343449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 714.368942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 714.414440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 714.453984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 714.466499] LNet: 58933:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 714.478591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 714.502771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 714.528860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 714.559907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 714.605387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 714.630647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 714.653504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 715.027992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp -m GET -e local_no_route -r 1 [ 715.067981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp -m GET -e local_no_route -r 1 [ 715.099390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.152@tcp [ 715.273396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 715.290339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 715.310790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 715.334953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 715.354814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 715.380604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 715.404779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 715.429830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 715.447840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 715.470759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 715.506513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 715.529314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 715.570355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 715.595897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 715.629401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 716.010979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp -m GET -e local_timeout -r 1 [ 716.039898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp -m GET -e local_timeout -r 1 [ 716.072265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.152@tcp [ 716.084753] LNet: There was an unexpected network error while writing to 192.168.202.152: rc = -22 [ 716.088477] LNet: Skipped 2 previous similar messages [ 716.237522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 716.255643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 716.280836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 716.306915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 716.327798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 716.347799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 716.375838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 716.401271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 716.427974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 716.454298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 716.477813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 716.501758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 716.523619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 716.544627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 716.566360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 716.869907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp -m GET -e local_error -r 1 [ 716.888228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp -m GET -e local_error -r 1 [ 716.908564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.152@tcp [ 717.089069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 717.112930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 717.142581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 717.171952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 717.206595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 717.244554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 717.267363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 717.287522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 717.300574] LNet: 59260:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 717.305550] LNet: 59260:0:(api-ni.c:296:recovery_interval_set()) Skipped 2 previous similar messages [ 717.320776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 717.350896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 717.371456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 717.393862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 717.416736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 717.435077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 717.454837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 717.631839] LNet: 59301:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 717.640257] LNetError: 59301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 717.658200] LNet: Removed LNI 192.168.202.52@tcp [ 717.985938] Key type .llcrypt unregistered [ 717.988722] Key type ._llcrypt unregistered [ 728.771944] Key type ._llcrypt registered [ 728.775863] Key type .llcrypt registered [ 729.589227] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 21:51:23 (1789264283) [ 730.047453] Key type .llcrypt unregistered [ 730.049862] Key type ._llcrypt unregistered [ 733.685951] Key type ._llcrypt registered [ 733.687955] Key type .llcrypt registered [ 733.827030] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 733.834807] alg: No test for adler32 (adler32-zlib) [ 734.708595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 734.731704] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 739.757919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.152@tcp [ 746.217649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 746.240251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 746.270297] LNet: Added LNI 192.168.202.52@tcp1 [8/256/0/180] [ 747.176670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp -m GET -e local_interrupt -r 1 [ 747.203555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp1 -m GET -e local_interrupt -r 1 [ 747.230164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp -m GET -e local_interrupt -r 1 [ 747.251462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp1 -m GET -e local_interrupt -r 1 [ 747.275215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp1 -d 192.168.202.152@tcp -m GET -e local_interrupt -r 1 [ 747.300886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp1 -d 192.168.202.152@tcp1 -m GET -e local_interrupt -r 1 [ 747.334884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp1 -d 192.168.202.52@tcp -m GET -e local_interrupt -r 1 [ 747.369515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp1 -d 192.168.202.52@tcp1 -m GET -e local_interrupt -r 1 [ 747.404129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.152@tcp [ 747.417822] LNet: There was an unexpected network error while writing to 192.168.202.152: rc = -22 [ 747.422336] LNet: 1 local NIs in recovery (showing 1): 192.168.202.52@tcp [ 747.618588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 747.639857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 747.663509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 747.686347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 747.703546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 747.722041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 747.738779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 747.756892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 747.771545] LNet: 60761:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 747.785339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 747.811779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 747.849566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 747.880622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 747.921881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 747.951769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 747.992288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 748.439185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp -m GET -e local_dropped -r 1 [ 748.467317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp1 -m GET -e local_dropped -r 1 [ 748.495162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp -m GET -e local_dropped -r 1 [ 748.513991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp1 -m GET -e local_dropped -r 1 [ 748.541097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp1 -d 192.168.202.152@tcp -m GET -e local_dropped -r 1 [ 748.569853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp1 -d 192.168.202.152@tcp1 -m GET -e local_dropped -r 1 [ 748.602773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp1 -d 192.168.202.52@tcp -m GET -e local_dropped -r 1 [ 748.646146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp1 -d 192.168.202.52@tcp1 -m GET -e local_dropped -r 1 [ 748.680605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.152@tcp [ 748.701617] LNet: There was an unexpected network error while writing to 192.168.202.152: rc = -22 [ 748.706447] LNet: Skipped 2 previous similar messages [ 748.896821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 748.922520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 748.947542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 748.972689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 749.003840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 749.034489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 749.066320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 749.098567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 749.113631] LNet: 60882:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 749.132049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 749.159670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 749.186172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 749.217526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 749.241839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 749.271733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 749.295920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 749.740349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp -m GET -e local_aborted -r 1 [ 749.767400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp1 -m GET -e local_aborted -r 1 [ 749.798734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp -m GET -e local_aborted -r 1 [ 749.837508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp1 -m GET -e local_aborted -r 1 [ 749.865473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp1 -d 192.168.202.152@tcp -m GET -e local_aborted -r 1 [ 749.899241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp1 -d 192.168.202.152@tcp1 -m GET -e local_aborted -r 1 [ 749.932056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp1 -d 192.168.202.52@tcp -m GET -e local_aborted -r 1 [ 749.967433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp1 -d 192.168.202.52@tcp1 -m GET -e local_aborted -r 1 [ 750.008138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.152@tcp [ 750.029150] LNet: There was an unexpected network error while writing to 192.168.202.152: rc = -22 [ 750.036195] LNet: Skipped 2 previous similar messages [ 750.231234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 750.255493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 750.282291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 750.303522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 750.325683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 750.358215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 750.382731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 750.401280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 750.415304] LNet: 61003:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 750.446354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 750.471599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 750.494703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 750.529349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 750.564896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 750.586981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 750.622700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 751.043153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp -m GET -e local_no_route -r 1 [ 751.068339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp1 -m GET -e local_no_route -r 1 [ 751.102300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp -m GET -e local_no_route -r 1 [ 751.129688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp1 -m GET -e local_no_route -r 1 [ 751.160444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp1 -d 192.168.202.152@tcp -m GET -e local_no_route -r 1 [ 751.190508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp1 -d 192.168.202.152@tcp1 -m GET -e local_no_route -r 1 [ 751.225412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp1 -d 192.168.202.52@tcp -m GET -e local_no_route -r 1 [ 751.255565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp1 -d 192.168.202.52@tcp1 -m GET -e local_no_route -r 1 [ 751.292365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.152@tcp [ 751.469454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 751.492097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 751.512410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 751.529001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 751.546637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 751.572814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 751.602562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 751.635855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 751.662131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 751.689900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 751.709267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 751.729406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 751.754238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 751.784683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 751.817923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 752.200849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp -m GET -e local_timeout -r 1 [ 752.232809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp1 -m GET -e local_timeout -r 1 [ 752.263235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp -m GET -e local_timeout -r 1 [ 752.290659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp1 -m GET -e local_timeout -r 1 [ 752.316415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp1 -d 192.168.202.152@tcp -m GET -e local_timeout -r 1 [ 752.345993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp1 -d 192.168.202.152@tcp1 -m GET -e local_timeout -r 1 [ 752.374286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp1 -d 192.168.202.52@tcp -m GET -e local_timeout -r 1 [ 752.402772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp1 -d 192.168.202.52@tcp1 -m GET -e local_timeout -r 1 [ 752.449924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.152@tcp [ 752.465111] LNet: There was an unexpected network error while writing to 192.168.202.152: rc = -22 [ 752.469593] LNet: Skipped 5 previous similar messages [ 752.621959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 752.641348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 752.661127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 752.685715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 752.716491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 752.745332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 752.771608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 752.794869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 752.815094] LNet: 61245:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 752.825345] LNet: 61245:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 752.841679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 752.861682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 752.880438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 752.901607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 752.925934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 752.957761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 752.988587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 753.374778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp -m GET -e local_error -r 1 [ 753.404736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp1 -m GET -e local_error -r 1 [ 753.437540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp -m GET -e local_error -r 1 [ 753.466919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp1 -m GET -e local_error -r 1 [ 753.497487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp1 -d 192.168.202.152@tcp -m GET -e local_error -r 1 [ 753.529605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp1 -d 192.168.202.152@tcp1 -m GET -e local_error -r 1 [ 753.556247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp1 -d 192.168.202.52@tcp -m GET -e local_error -r 1 [ 753.588236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp1 -d 192.168.202.52@tcp1 -m GET -e local_error -r 1 [ 753.622046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.152@tcp [ 753.791167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 753.815884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 753.837646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 753.856460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 753.872966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 753.889198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 753.911212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 753.937539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 753.959360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 753.989749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 754.013569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 754.035466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 754.062164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 754.085549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 754.106486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 757.551850] LNet: 61442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 757.561971] LNetError: 61442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 757.565771] LNet: Removed LNI 192.168.202.52@tcp [ 757.576610] LNet: Removed LNI 192.168.202.52@tcp1 [ 757.832723] Key type .llcrypt unregistered [ 757.835267] Key type ._llcrypt unregistered [ 768.834467] Key type ._llcrypt registered [ 768.836096] Key type .llcrypt registered [ 769.577973] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 21:52:03 (1789264323) [ 770.079505] Key type .llcrypt unregistered [ 770.081595] Key type ._llcrypt unregistered [ 774.066860] Key type ._llcrypt registered [ 774.069920] Key type .llcrypt registered [ 774.188502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 774.198903] alg: No test for adler32 (adler32-zlib) [ 775.063716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 775.086357] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 779.643911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.152@tcp [ 779.995784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp -m GET -e remote_dropped -r 1 [ 780.015608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp -m GET -e remote_dropped -r 1 [ 780.036578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.152@tcp [ 780.053215] LNet: There was an unexpected network error while writing to 192.168.202.152: rc = -22 [ 780.164748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 780.180438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 780.197805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 780.215939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 780.237890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 780.258836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 780.276935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 780.293479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 780.304621] LNet: 62684:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 780.314702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 780.332800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 780.349652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 780.366150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 780.385754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 780.404748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 780.426456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 780.739687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp -m GET -e remote_error -r 1 [ 780.777461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp -m GET -e remote_error -r 1 [ 780.798989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.152@tcp [ 780.811889] LNet: There was an unexpected network error while writing to 192.168.202.152: rc = -22 [ 780.964559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 780.997251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 781.032080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 781.055369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 781.079484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 781.103409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 781.137248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 781.171659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 781.183301] LNet: 62793:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 781.199980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 781.230874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 781.255328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 781.288447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 781.315418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 781.346241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 781.378314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 781.722806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp -m GET -e remote_timeout -r 1 [ 781.763099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp -m GET -e remote_timeout -r 1 [ 781.796483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.152@tcp [ 781.960647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 781.982705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 782.006544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 782.033833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 782.060550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 782.083664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 782.106545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 782.129532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 782.160845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 782.190480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 782.213418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 782.230992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 782.258958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 782.284618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 782.312247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 782.450175] LNet: 62944:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 782.454934] LNetError: 62944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 782.463582] LNet: Removed LNI 192.168.202.52@tcp [ 782.682696] Key type .llcrypt unregistered [ 782.685271] Key type ._llcrypt unregistered [ 792.778416] Key type ._llcrypt registered [ 792.780510] Key type .llcrypt registered [ 793.727670] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 21:52:27 (1789264347) [ 794.279378] Key type .llcrypt unregistered [ 794.282165] Key type ._llcrypt unregistered [ 797.972936] Key type ._llcrypt registered [ 797.975356] Key type .llcrypt registered [ 798.189866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 798.200741] alg: No test for adler32 (adler32-zlib) [ 799.111308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 799.138554] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 805.195700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.152@tcp [ 812.669387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 812.709611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 812.723846] LNet: Added LNI 192.168.202.52@tcp1 [8/256/0/180] [ 813.812391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp -m GET -e remote_dropped -r 1 [ 813.847800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp1 -m GET -e remote_dropped -r 1 [ 813.888988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp -m GET -e remote_dropped -r 1 [ 813.921660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp1 -m GET -e remote_dropped -r 1 [ 813.954202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp1 -d 192.168.202.152@tcp -m GET -e remote_dropped -r 1 [ 813.992693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp1 -d 192.168.202.152@tcp1 -m GET -e remote_dropped -r 1 [ 814.021901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp1 -d 192.168.202.52@tcp -m GET -e remote_dropped -r 1 [ 814.047193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp1 -d 192.168.202.52@tcp1 -m GET -e remote_dropped -r 1 [ 814.070986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.152@tcp [ 814.089588] LNet: There was an unexpected network error while writing to 192.168.202.152: rc = -22 [ 814.098158] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.152@tcp [ 814.273389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 814.297995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 814.327105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 814.354364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 814.387093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 814.424599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 814.450664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 814.482273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 814.497381] LNet: 64402:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 814.527342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 814.551789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 814.572564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 814.599799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 814.623930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 814.659104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 814.696390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 814.729415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 815.161364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp -m GET -e remote_error -r 1 [ 815.194254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp1 -m GET -e remote_error -r 1 [ 815.224427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp -m GET -e remote_error -r 1 [ 815.261453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp1 -m GET -e remote_error -r 1 [ 815.299298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp1 -d 192.168.202.152@tcp -m GET -e remote_error -r 1 [ 815.322967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp1 -d 192.168.202.152@tcp1 -m GET -e remote_error -r 1 [ 815.353919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp1 -d 192.168.202.52@tcp -m GET -e remote_error -r 1 [ 815.389649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp1 -d 192.168.202.52@tcp1 -m GET -e remote_error -r 1 [ 815.436697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.152@tcp [ 815.460219] LNet: There was an unexpected network error while writing to 192.168.202.152: rc = -22 [ 815.465484] LNet: Skipped 2 previous similar messages [ 815.682588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 815.712518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 815.737926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 815.768049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 815.788938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 815.815696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 815.846944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 815.893262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 815.914290] LNet: 64525:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 815.944096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 815.971902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 815.991171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 816.012775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 816.044953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 816.067556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 816.093485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 816.141064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 816.606889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp -m GET -e remote_timeout -r 1 [ 816.637921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp1 -m GET -e remote_timeout -r 1 [ 816.685381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp -m GET -e remote_timeout -r 1 [ 816.727562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp1 -m GET -e remote_timeout -r 1 [ 816.759486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp1 -d 192.168.202.152@tcp -m GET -e remote_timeout -r 1 [ 816.797283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp1 -d 192.168.202.152@tcp1 -m GET -e remote_timeout -r 1 [ 816.829349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp1 -d 192.168.202.52@tcp -m GET -e remote_timeout -r 1 [ 816.862890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp1 -d 192.168.202.52@tcp1 -m GET -e remote_timeout -r 1 [ 816.895972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.152@tcp [ 816.917425] LNet: There was an unexpected network error while writing to 192.168.202.152: rc = -22 [ 817.108424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 817.124624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 817.144180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 817.170183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 817.200990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 817.241043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 817.281157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 817.316202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 817.340695] LNet: 64649:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 817.381266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 817.415866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 817.442803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 817.476925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 817.500376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 817.532577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 817.558963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 817.589660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 822.880539] LNet: 64726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 822.883741] LNetError: 64726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 823.904260] LNet: Removed LNI 192.168.202.52@tcp [ 823.908170] LNetError: 64726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 823.932579] LNet: Removed LNI 192.168.202.52@tcp1 [ 824.414380] Key type .llcrypt unregistered [ 824.416667] Key type ._llcrypt unregistered [ 836.879489] Key type ._llcrypt registered [ 836.886118] Key type .llcrypt registered [ 837.805901] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 21:53:11 (1789264391) [ 838.816373] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 839.974191] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 21:53:14 (1789264394) [ 840.604511] Key type .llcrypt unregistered [ 840.606188] Key type ._llcrypt unregistered [ 844.881928] Key type ._llcrypt registered [ 844.884221] Key type .llcrypt registered [ 845.008388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 845.026778] alg: No test for adler32 (adler32-zlib) [ 845.956174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 845.976080] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 851.554914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.152@tcp [ 852.144713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp -m GET -e network_timeout -r 1 [ 852.201993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp -m GET -e network_timeout -r 1 [ 852.253675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.152@tcp [ 852.279926] LNet: There was an unexpected network error while writing to 192.168.202.152: rc = -22 [ 852.426610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 852.446206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 852.463800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 852.485476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 852.503305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 852.518621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 852.544634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 852.566544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 852.585422] LNet: 66263:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 852.602565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 852.623461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 852.641849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 852.662676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 852.694523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 852.726809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 852.905509] LNet: 66302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 852.909743] LNetError: 66302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 853.930071] LNet: Removed LNI 192.168.202.52@tcp [ 854.250683] Key type .llcrypt unregistered [ 854.253046] Key type ._llcrypt unregistered [ 865.933402] Key type ._llcrypt registered [ 865.935815] Key type .llcrypt registered [ 866.078207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 866.088389] alg: No test for adler32 (adler32-zlib) [ 866.980358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 866.996861] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 871.964264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.152@tcp [ 877.802456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 877.827205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 877.842403] LNet: Added LNI 192.168.202.52@tcp1 [8/256/0/180] [ 878.742574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp -m GET -e network_timeout -r 1 [ 878.774615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp1 -m GET -e network_timeout -r 1 [ 878.807480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp -m GET -e network_timeout -r 1 [ 878.838642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp1 -m GET -e network_timeout -r 1 [ 878.870667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp1 -d 192.168.202.152@tcp -m GET -e network_timeout -r 1 [ 878.903403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp1 -d 192.168.202.152@tcp1 -m GET -e network_timeout -r 1 [ 878.933579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp1 -d 192.168.202.52@tcp -m GET -e network_timeout -r 1 [ 878.967907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp1 -d 192.168.202.52@tcp1 -m GET -e network_timeout -r 1 [ 878.988814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.152@tcp [ 879.002226] LNet: There was an unexpected network error while writing to 192.168.202.152: rc = -22 [ 879.171703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 879.199487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 879.225443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 879.254793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 879.280166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 879.307934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 879.329166] LNet: 1 local NIs in recovery (showing 1): 192.168.202.52@tcp [ 879.333032] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.152@tcp [ 879.334090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 879.362813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 879.374831] LNet: 67174:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 879.389496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 879.409986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 879.432680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 879.461284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 879.489208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 879.519952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 882.602170] LNet: There was an unexpected network error while writing to 192.168.202.152: rc = -22 [ 882.807556] LNet: 67247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.812551] LNetError: 67247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 882.817318] LNet: Removed LNI 192.168.202.52@tcp [ 882.832649] LNet: Removed LNI 192.168.202.52@tcp1 [ 883.059581] Key type .llcrypt unregistered [ 883.061199] Key type ._llcrypt unregistered [ 892.275746] Key type ._llcrypt registered [ 892.277360] Key type .llcrypt registered [ 893.070484] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 21:54:07 (1789264447) [ 893.369519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 893.380373] alg: No test for adler32 (adler32-zlib) [ 894.241404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 894.280636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 894.296872] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 894.783815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 894.800131] LNet: Added LNI 192.168.202.52@tcp1 [8/256/0/180] [ 895.507529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.52@tcp [ 895.571346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 895.608526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 895.632750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 895.655604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 895.679600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 895.712780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.52@tcp [ 895.730423] LNetError: 68343:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.52@tcp to recovery [ 896.288963] LNet: 1 local NIs in recovery (showing 1): 192.168.202.52@tcp [ 896.289214] LNet: There was an unexpected network error while writing to 192.168.202.52: rc = -22 [ 898.341388] LNet: There was an unexpected network error while writing to 192.168.202.52: rc = -22 [ 899.142222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 902.434832] LNet: There was an unexpected network error while writing to 192.168.202.52: rc = -22 [ 902.527179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 902.547230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 902.591311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 902.609577] LNet: 68450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 902.614487] LNetError: 68450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 903.648242] LNet: Removed LNI 192.168.202.52@tcp [ 903.651322] LNetError: 68450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 903.664129] LNet: Removed LNI 192.168.202.52@tcp1 [ 903.678967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 903.729467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 903.745338] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 904.205285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 904.216990] LNet: Added LNI 192.168.202.52@tcp1 [8/256/0/180] [ 904.685837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.52@tcp [ 904.708684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 904.742766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 904.777790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 904.798754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 904.823475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 904.841451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 904.870105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.52@tcp [ 904.891891] LNetError: 68722:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.52@tcp to recovery [ 905.760457] LNet: There was an unexpected network error while writing to 192.168.202.52: rc = -22 [ 908.218896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 911.904742] LNet: There was an unexpected network error while writing to 192.168.202.52: rc = -22 [ 911.909245] LNet: Skipped 1 previous similar message [ 917.017676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 917.041025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 917.073416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 921.903609] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 21:54:36 (1789264476) [ 921.979558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 922.002681] LNet: 69445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 922.006944] LNetError: 69445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 922.012524] LNet: Removed LNI 192.168.202.52@tcp [ 922.022672] LNet: Removed LNI 192.168.202.52@tcp1 [ 922.052078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 922.098633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 922.113678] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 922.570545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 922.579535] LNet: Added LNI 192.168.202.52@tcp1 [8/256/0/180] [ 923.085368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.52@tcp [ 923.141822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 923.162636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 923.186537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 923.214652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 923.242373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 923.266356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.52@tcp --health 0 [ 923.334697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 924.128362] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.52@tcp [ 924.128838] LNet: There was an unexpected network error while writing to 192.168.202.52: rc = -22 [ 924.136553] LNet: Skipped 1 previous similar message [ 939.325387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 939.422402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 939.441979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 939.477437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.52@tcp --health 500 [ 939.581284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 946.076914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 946.111932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 946.125592] LNet: 69951:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 946.129187] LNetError: 69951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 946.133303] LNetError: 69951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 946.137342] LNet: Removed LNI 192.168.202.52@tcp [ 946.145978] LNet: Removed LNI 192.168.202.52@tcp1 [ 946.167605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 946.202855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 946.215293] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 946.665079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 946.676701] LNet: Added LNI 192.168.202.52@tcp1 [8/256/0/180] [ 947.179444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.52@tcp [ 947.206085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 947.238962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 947.255697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 947.271669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 947.295716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 947.316316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 947.340207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.52@tcp --health 0 [ 948.257618] LNet: There was an unexpected network error while writing to 192.168.202.52: rc = -22 [ 948.264218] LNet: Skipped 3 previous similar messages [ 948.532703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 959.260996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 959.278217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 959.302695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 963.091763] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 21:55:17 (1789264517) [ 963.349963] LNet: 71008:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 963.354749] LNetError: 71008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 963.361598] LNetError: 71008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 963.370969] LNet: Removed LNI 192.168.202.52@tcp [ 963.381236] LNet: Removed LNI 192.168.202.52@tcp1 [ 963.778930] Key type .llcrypt unregistered [ 963.781636] Key type ._llcrypt unregistered [ 966.946710] Key type ._llcrypt registered [ 966.948983] Key type .llcrypt registered [ 967.059969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 967.072860] alg: No test for adler32 (adler32-zlib) [ 967.918305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 967.936561] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 972.263417] Lustre: DEBUG MARKER: Initial discovery [ 972.317835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.152@tcp [ 973.359849] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 973.393158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 973.430860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.152@tcp [ 973.458654] LNet: There was an unexpected network error while writing to 192.168.202.152: rc = -22 [ 973.476848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 973.505677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.152@tcp -r 1 -m GET -l 3 [ 973.523144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.152@tcp -r 1 -m GET -e local_error [ 973.548035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.152@tcp -r 1 -m PUT -l 6 [ 974.113353] LNet: 1 local NIs in recovery (showing 1): 192.168.202.52@tcp [ 974.638435] Lustre: DEBUG MARKER: Force oleg252-client.virtnet to discover 192.168.202.152@tcp (in background) [ 974.688154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.152@tcp [ 975.619264] Lustre: DEBUG MARKER: Force oleg252-server to discover 192.168.202.52@tcp [ 977.091079] Lustre: DEBUG MARKER: Wait for 71759 [ 977.760314] LNet: There was an unexpected network error while writing to 192.168.202.152: rc = -22 [ 978.582352] Lustre: DEBUG MARKER: Finished wait on 71759 [ 978.603125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 978.631570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 978.805603] LNet: 72221:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 978.809972] LNetError: 72221:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 979.882055] LNet: Removed LNI 192.168.202.52@tcp [ 980.080454] Key type .llcrypt unregistered [ 980.082152] Key type ._llcrypt unregistered [ 989.314795] Key type ._llcrypt registered [ 989.315968] Key type .llcrypt registered [ 990.159201] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 21:55:44 (1789264544) [ 990.607490] Key type .llcrypt unregistered [ 990.609288] Key type ._llcrypt unregistered [ 994.800330] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 994.877829] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 994.883715] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 994.891917] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 997.311130] Key type ._llcrypt registered [ 997.312326] Key type .llcrypt registered [ 997.431458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 997.442710] alg: No test for adler32 (adler32-zlib) [ 998.319230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 998.394449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 998.430235] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 999.086543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 999.106129] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1004.952272] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 21:55:59 (1789264559) [ 1005.133369] LNet: 74293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1005.136662] LNetError: 74293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1005.143792] LNet: Removed LNI 10.1.2.3@tcp [ 1005.155162] LNet: Removed LNI 192.168.202.52@tcp [ 1005.587395] Key type .llcrypt unregistered [ 1005.589692] Key type ._llcrypt unregistered [ 1009.474311] Key type ._llcrypt registered [ 1009.476303] Key type .llcrypt registered [ 1009.607325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1009.614294] alg: No test for adler32 (adler32-zlib) [ 1010.463568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1010.489957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1010.503044] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1010.883339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1010.904415] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1011.511085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.52@tcp [ 1012.577164] LNetError: Unexpected error -22 connecting to 192.168.202.52@tcp at host 192.168.202.52:988 [ 1014.908811] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 21:56:09 (1789264569) [ 1014.960246] LNet: 75412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1014.966176] LNetError: 75412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1016.032462] LNet: Removed LNI 10.1.2.3@tcp [ 1016.035486] LNetError: 75412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1016.053396] LNet: Removed LNI 192.168.202.52@tcp [ 1016.534824] Key type .llcrypt unregistered [ 1016.537633] Key type ._llcrypt unregistered [ 1020.145585] Key type ._llcrypt registered [ 1020.147897] Key type .llcrypt registered [ 1020.256178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1020.270544] alg: No test for adler32 (adler32-zlib) [ 1021.122157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1021.156909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1021.170731] LNet: Added LNI 192.168.202.52@tcp1 [8/256/0/180] [ 1021.659452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1021.670069] LNet: Added LNI 192.168.202.52@tcp2 [8/256/0/180] [ 1022.200532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.52@tcp1 --nid 192.168.202.52@tcp2 [ 1024.797570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.52@tcp1 192.168.202.52@tcp2 [ 1024.808387] LNetError: 76253:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.52@tcp1 to 192.168.202.52@tcp2 - Route table may be misconfigured [ 1024.820068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.52@tcp2 192.168.202.52@tcp1 [ 1028.497966] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 21:56:22 (1789264582) [ 1028.588285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1028.601987] LNet: 76826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1028.606863] LNetError: 76826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1028.610397] LNet: Removed LNI 192.168.202.52@tcp1 [ 1029.677241] LNet: Removed LNI 192.168.202.52@tcp2 [ 1029.743706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1029.778131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1029.796900] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1030.233808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1030.249070] LNet: Added LNI 192.168.202.52@tcp1 [8/256/0/180] [ 1030.811350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.52@tcp [ 1030.835433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.52@tcp 192.168.202.52@tcp [ 1030.872619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.52@tcp1 192.168.202.52@tcp1 [ 1030.904329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.52@tcp -d 192.168.202.52@tcp -e network_timeout [ 1030.930716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.52@tcp -d 192.168.202.52@tcp1 -e network_timeout [ 1030.959412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.52@tcp1 -d 192.168.202.52@tcp -e network_timeout [ 1030.988057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.52@tcp1 -d 192.168.202.52@tcp1 -e network_timeout [ 1031.006520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.52@tcp [ 1031.017062] LNet: There was an unexpected network error while writing to 192.168.202.52: rc = -22 [ 1031.030372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1031.092072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1031.168496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1031.841084] LNet: 1 local NIs in recovery (showing 1): 192.168.202.52@tcp [ 1034.906835] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 21:56:29 (1789264589) [ 1035.083874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1035.105898] LNet: 77687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1035.116859] LNetError: 77687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1035.122137] LNetError: 77687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1035.129330] LNet: Removed LNI 192.168.202.52@tcp [ 1036.199814] LNet: Removed LNI 192.168.202.52@tcp1 [ 1036.215287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1036.258695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1036.413847] LNet: 77724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1036.653758] Key type .llcrypt unregistered [ 1036.654765] Key type ._llcrypt unregistered [ 1043.381642] Key type ._llcrypt registered [ 1043.383209] Key type .llcrypt registered [ 1044.141933] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 21:56:38 (1789264598) [ 1044.335722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1044.343324] alg: No test for adler32 (adler32-zlib) [ 1045.190572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1045.893888] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1046.930629] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 21:56:41 (1789264601) [ 1046.997428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1047.010437] LNet: 78816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1047.023429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1047.064956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1047.079733] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1047.498821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1047.513997] LNet: Added LNI 192.168.202.52@tcp1 [8/256/0/180] [ 1047.947668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.52@tcp [ 1047.968451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.52@tcp1 [ 1047.990569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.52@tcp1 [ 1051.490501] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 21:56:45 (1789264605) [ 1052.556114] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg252-server [ 1053.522668] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 21:56:47 (1789264607) [ 1054.629395] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg252-server [ 1055.792625] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 21:56:50 (1789264610) [ 1057.027305] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg252-server [ 1058.039388] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 21:56:52 (1789264612) [ 1059.336456] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg252-server [ 1060.528098] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 21:56:54 (1789264614) [ 1061.696348] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg252-server [ 1062.886754] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 21:56:57 (1789264617) [ 1064.047218] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg252-server [ 1065.203233] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 21:56:59 (1789264619) [ 1066.352451] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg252-server [ 1067.390475] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 21:57:01 (1789264621) [ 1068.501411] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg252-server [ 1069.602996] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 21:57:03 (1789264623) [ 1070.595387] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg252-server [ 1071.636512] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 21:57:06 (1789264626) [ 1071.838644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1071.850633] LNet: 82692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1071.854910] LNetError: 82692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1071.859944] LNet: Removed LNI 192.168.202.52@tcp [ 1072.872435] LNet: Removed LNI 192.168.202.52@tcp1 [ 1072.900459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1072.944342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1072.962618] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1073.479912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1073.554886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.52@tcp [ 1073.692623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1073.706257] LNet: 82840:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1073.709855] LNetError: 82840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1073.713202] LNetError: 82840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1074.726609] LNet: Removed LNI 192.168.202.52@tcp [ 1074.745705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1074.780779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1074.802834] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1075.237979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.52@tcp [ 1075.358175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1075.369903] LNetError: 82985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1075.378428] LNet: Removed LNI 192.168.202.52@tcp [ 1075.398133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1075.455308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1075.473426] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1075.912488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1075.972565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.52@tcp [ 1076.125918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1076.143829] LNet: 83132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1076.146649] LNet: 83132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1077.159456] LNet: Removed LNI 192.168.202.52@tcp [ 1077.175440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1077.218130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1077.239152] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1077.761583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.52@tcp [ 1077.889330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1077.922755] LNetError: 83277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1077.929546] LNetError: 83277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1077.943922] LNet: Removed LNI 192.168.202.52@tcp [ 1077.966907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1078.019264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1078.044768] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1078.469082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1078.519744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.52@tcp [ 1078.646171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1079.722128] LNet: Removed LNI 192.168.202.52@tcp [ 1079.745637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1079.785378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1079.812903] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1080.239404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.52@tcp [ 1080.390566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1080.410240] LNet: 83570:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1080.413151] LNet: 83570:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1081.453167] LNet: Removed LNI 192.168.202.52@tcp [ 1081.490934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1081.551190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1081.576515] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1082.160474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1082.219820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.52@tcp [ 1082.374178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1082.395553] LNetError: 83717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1082.404662] LNetError: 83717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1082.425721] LNet: Removed LNI 192.168.202.52@tcp [ 1082.440395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1082.477234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1082.504110] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1083.104916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.52@tcp [ 1083.249149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1084.328203] LNet: Removed LNI 192.168.202.52@tcp [ 1084.352123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1084.408381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1084.422589] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1084.869893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1084.945928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.52@tcp [ 1087.299994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1088.360887] LNet: Removed LNI 192.168.202.52@tcp [ 1088.376428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1088.417988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1088.435443] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1088.968838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.52@tcp [ 1090.220180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1090.236785] LNet: 84182:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1090.241755] LNet: 84182:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1091.314975] LNet: Removed LNI 192.168.202.52@tcp [ 1091.349976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1091.392395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1091.408410] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1091.829266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1091.875315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.52@tcp [ 1092.017793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1092.036770] LNetError: 84329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1092.044145] LNetError: 84329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1092.053332] LNet: Removed LNI 192.168.202.52@tcp [ 1092.069434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1092.112781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1092.131710] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1092.612453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.52@tcp [ 1092.835373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1093.862675] LNet: Removed LNI 192.168.202.52@tcp [ 1093.884990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1093.924815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1093.945084] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1094.401809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1094.446504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1094.477250] LNet: Removed LNI 192.168.202.52@tcp [ 1094.489830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1094.511625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1094.526182] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1095.046434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1095.096919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1095.116169] LNet: Removed LNI 192.168.202.52@tcp [ 1095.130448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1095.167709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1095.225733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1095.253858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1095.290586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1095.344102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1095.379314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1095.423989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1095.490897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1095.515965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1095.552209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1095.568439] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1096.166326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1096.188852] LNet: Removed LNI 192.168.202.52@tcp [ 1096.202755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1096.259555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1100.859794] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 21:57:35 (1789264655) [ 1100.920289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1100.952778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1100.998309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1101.017991] LNet: Added LNI 192.168.202.52@tcp231 [8/256/0/180] [ 1101.816707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1101.842493] LNet: Removed LNI 192.168.202.52@tcp231 [ 1101.853719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1101.894798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1101.911826] LNet: Added LNI 192.168.202.52@tcp231 [8/256/0/180] [ 1101.916684] LNet: 85641:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1102.268658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1102.293177] LNet: Removed LNI 192.168.202.52@tcp231 [ 1102.312765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1102.336459] LNet: Added LNI 192.168.202.52@tcp231 [8/256/0/180] [ 1102.975189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1102.999936] LNet: Removed LNI 192.168.202.52@tcp231 [ 1103.010526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1103.065388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1103.086924] LNet: Added LNI 192.168.202.52@tcp231 [8/256/0/0] [ 1103.091161] LNet: 85793:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1103.340521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1103.370910] LNet: Removed LNI 192.168.202.52@tcp231 [ 1103.383737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1103.404649] LNet: Added LNI 192.168.202.52@tcp231 [8/256/0/0] [ 1104.153153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1104.175592] LNet: Removed LNI 192.168.202.52@tcp231 [ 1104.190441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1104.230470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1104.246388] LNet: Added LNI 192.168.202.52@tcp231 [8/256/0/60] [ 1104.250874] LNet: 85944:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1104.472270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1104.500092] LNet: Removed LNI 192.168.202.52@tcp231 [ 1104.509559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1104.532909] LNet: Added LNI 192.168.202.52@tcp231 [8/256/0/60] [ 1105.295430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1105.317599] LNet: Removed LNI 192.168.202.52@tcp231 [ 1105.327864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1105.366371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1105.387924] LNet: Added LNI 192.168.202.52@tcp231 [8/256/0/180] [ 1105.622705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1105.652861] LNet: Removed LNI 192.168.202.52@tcp231 [ 1105.672469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1105.696922] LNet: Added LNI 192.168.202.52@tcp231 [8/256/0/180] [ 1109.713473] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 21:57:44 (1789264664) [ 1109.827419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1109.856457] LNet: 86805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1109.860587] LNet: 86805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 13 previous similar messages [ 1109.870963] LNetError: 86805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1109.874620] LNetError: 86805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 1109.886820] LNet: Removed LNI 192.168.202.52@tcp231 [ 1109.900365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1109.936810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1109.958568] LNet: Added LNI 192.168.202.52@tcp232 [8/256/0/180] [ 1110.457672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1110.483915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1110.513457] LNet: Removed LNI 192.168.202.52@tcp232 [ 1110.527734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1110.543134] LNet: Added LNI 192.168.202.52@tcp232 [8/256/0/180] [ 1110.951223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1110.982666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1111.004646] LNet: Removed LNI 192.168.202.52@tcp232 [ 1111.021672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1111.037148] LNet: Added LNI 192.168.202.52@tcp232 [8/256/0/180] [ 1111.492504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1111.525973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1111.550840] LNet: Removed LNI 192.168.202.52@tcp232 [ 1111.566385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1111.586722] LNet: Added LNI 192.168.202.52@tcp232 [8/256/0/180] [ 1112.075234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1112.104336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1112.132971] LNet: Removed LNI 192.168.202.52@tcp232 [ 1112.146067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1116.267923] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 21:57:50 (1789264670) [ 1117.005984] Key type .llcrypt unregistered [ 1117.013171] Key type ._llcrypt unregistered [ 1120.499243] Key type ._llcrypt registered [ 1120.502818] Key type .llcrypt registered [ 1120.666356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1120.678742] alg: No test for adler32 (adler32-zlib) [ 1121.552701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1121.572588] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1126.245629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.152@tcp [ 1130.981568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1131.009188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1131.032817] LNet: Added LNI 192.168.202.52@tcp1 [8/256/0/180] [ 1131.805958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp -m GET -e local_interrupt -r 2 [ 1131.830871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp1 -m GET -e local_interrupt -r 2 [ 1131.861146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp -m GET -e local_interrupt -r 2 [ 1131.882174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp1 -m GET -e local_interrupt -r 2 [ 1131.921821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp1 -d 192.168.202.152@tcp -m GET -e local_interrupt -r 2 [ 1131.948182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp1 -d 192.168.202.152@tcp1 -m GET -e local_interrupt -r 2 [ 1131.971652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp1 -d 192.168.202.52@tcp -m GET -e local_interrupt -r 2 [ 1132.006415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp1 -d 192.168.202.52@tcp1 -m GET -e local_interrupt -r 2 [ 1132.034695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.52@tcp 192.168.202.152@tcp [ 1132.047731] LNet: There was an unexpected network error while writing to 192.168.202.152: rc = -22 [ 1132.054965] LNet: 1 local NIs in recovery (showing 1): 192.168.202.52@tcp [ 1132.176627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1132.198539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1132.221453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1132.240736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1132.265958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1132.282959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1132.301982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1132.327568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1132.342523] LNet: 88588:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1132.355259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1132.371731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1132.388763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1132.410146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1132.432767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1132.452182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1132.765449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp -m GET -e local_dropped -r 2 [ 1132.790132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp1 -m GET -e local_dropped -r 2 [ 1132.812691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp -m GET -e local_dropped -r 2 [ 1132.839719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp1 -m GET -e local_dropped -r 2 [ 1132.867698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp1 -d 192.168.202.152@tcp -m GET -e local_dropped -r 2 [ 1132.898191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp1 -d 192.168.202.152@tcp1 -m GET -e local_dropped -r 2 [ 1132.923806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp1 -d 192.168.202.52@tcp -m GET -e local_dropped -r 2 [ 1132.947894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp1 -d 192.168.202.52@tcp1 -m GET -e local_dropped -r 2 [ 1132.968359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.52@tcp 192.168.202.152@tcp [ 1132.983538] LNet: There was an unexpected network error while writing to 192.168.202.152: rc = -22 [ 1133.127602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1133.150775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1133.173717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1133.193367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1133.215697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1133.240724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1133.259053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1133.294719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1133.311934] LNet: 88708:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1133.326290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1133.344466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1133.368128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1133.389461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1133.411426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1133.429831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1133.732694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp -m GET -e local_aborted -r 2 [ 1133.754696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp1 -m GET -e local_aborted -r 2 [ 1133.780865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp -m GET -e local_aborted -r 2 [ 1133.808138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp1 -m GET -e local_aborted -r 2 [ 1133.835940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp1 -d 192.168.202.152@tcp -m GET -e local_aborted -r 2 [ 1133.869622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp1 -d 192.168.202.152@tcp1 -m GET -e local_aborted -r 2 [ 1133.896946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp1 -d 192.168.202.52@tcp -m GET -e local_aborted -r 2 [ 1133.920828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp1 -d 192.168.202.52@tcp1 -m GET -e local_aborted -r 2 [ 1133.943779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.52@tcp 192.168.202.152@tcp [ 1134.071371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1134.092162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1134.109109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1134.128649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1134.150386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1134.164585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1134.179418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1134.197439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1134.215740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1134.241803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1134.263051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1134.279331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1134.293438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1134.311555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1134.594231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp -m GET -e local_no_route -r 2 [ 1134.618948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp1 -m GET -e local_no_route -r 2 [ 1134.639417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp -m GET -e local_no_route -r 2 [ 1134.664495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp1 -m GET -e local_no_route -r 2 [ 1134.684855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp1 -d 192.168.202.152@tcp -m GET -e local_no_route -r 2 [ 1134.712132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp1 -d 192.168.202.152@tcp1 -m GET -e local_no_route -r 2 [ 1134.748807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp1 -d 192.168.202.52@tcp -m GET -e local_no_route -r 2 [ 1134.777934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp1 -d 192.168.202.52@tcp1 -m GET -e local_no_route -r 2 [ 1134.802686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.52@tcp 192.168.202.152@tcp [ 1134.817287] LNet: There was an unexpected network error while writing to 192.168.202.152: rc = -22 [ 1134.821034] LNet: Skipped 1 previous similar message [ 1134.977265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1134.997953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1135.018901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1135.036989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1135.053992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1135.081104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1135.109706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1135.129621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1135.137729] LNet: 88949:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1135.141087] LNet: 88949:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1135.154317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1135.169423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1135.190852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1135.209838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1135.239665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1135.266718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1135.581387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp -m GET -e local_timeout -r 2 [ 1135.606861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp1 -m GET -e local_timeout -r 2 [ 1135.634810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp -m GET -e local_timeout -r 2 [ 1135.657947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp1 -m GET -e local_timeout -r 2 [ 1135.678677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp1 -d 192.168.202.152@tcp -m GET -e local_timeout -r 2 [ 1135.699397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp1 -d 192.168.202.152@tcp1 -m GET -e local_timeout -r 2 [ 1135.719316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp1 -d 192.168.202.52@tcp -m GET -e local_timeout -r 2 [ 1135.742274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp1 -d 192.168.202.52@tcp1 -m GET -e local_timeout -r 2 [ 1135.762961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.52@tcp 192.168.202.152@tcp [ 1135.904546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1135.921996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1135.938958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1135.964766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1135.989981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1136.007814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1136.029473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1136.056472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1136.072394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1136.093569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1136.115910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1136.139934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1136.170969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1136.196962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1136.483565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp -m GET -e remote_dropped -r 2 [ 1136.500134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp -d 192.168.202.152@tcp1 -m GET -e remote_dropped -r 2 [ 1136.520534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp -m GET -e remote_dropped -r 2 [ 1136.542564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp -d 192.168.202.52@tcp1 -m GET -e remote_dropped -r 2 [ 1136.572796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp1 -d 192.168.202.152@tcp -m GET -e remote_dropped -r 2 [ 1136.593814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.52@tcp1 -d 192.168.202.152@tcp1 -m GET -e remote_dropped -r 2 [ 1136.622916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp1 -d 192.168.202.52@tcp -m GET -e remote_dropped -r 2 [ 1136.654251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.52@tcp1 -d 192.168.202.52@tcp1 -m GET -e remote_dropped -r 2 [ 1136.681320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.52@tcp 192.168.202.152@tcp [ 1136.698775] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.152@tcp [ 1136.843061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1136.862214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1136.881676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1136.905687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1136.930655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1136.953243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1136.982129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1137.007432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1137.036147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1137.059706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1137.084437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1137.112937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1137.147095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1137.175151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1140.270823] LNet: 89262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1140.273330] LNetError: 89262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1140.277625] LNet: Removed LNI 192.168.202.52@tcp [ 1140.286745] LNet: Removed LNI 192.168.202.52@tcp1 [ 1140.482970] Key type .llcrypt unregistered [ 1140.484611] Key type ._llcrypt unregistered [ 1149.370740] Key type ._llcrypt registered [ 1149.373641] Key type .llcrypt registered [ 1150.244692] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 21:58:24 (1789264704) [ 1150.471932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1150.488801] alg: No test for adler32 (adler32-zlib) [ 1151.348023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1151.389795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1151.411063] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1151.950659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1151.966124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.52@tcp [ 1151.974988] LNetError: 90236:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.52@tcp to recovery [ 1152.049820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.52@tcp [ 1152.130813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1155.442806] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 21:58:29 (1789264709) [ 1155.593287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1155.608368] LNet: 90827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1155.611738] LNetError: 90827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1155.630879] LNet: Removed LNI 192.168.202.52@tcp [ 1155.652881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1155.701221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1155.719773] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1156.129576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1156.150572] LNet: Added LNI 192.168.202.52@tcp1 [8/256/0/180] [ 1156.712127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.52@tcp [ 1156.735268] LNetError: 91093:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.52@tcp to recovery [ 1157.729618] LNet: 1 local NIs in recovery (showing 1): 192.168.202.52@tcp [ 1157.735644] LNet: There was an unexpected network error while writing to 192.168.202.52: rc = -22 [ 1157.871881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.52@tcp [ 1163.686175] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 21:58:38 (1789264718) [ 1164.593784] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg252-server [ 1165.563524] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 21:58:39 (1789264719) [ 1166.563036] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg252-server [ 1167.468035] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 21:58:41 (1789264721) [ 1168.528692] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg252-server [ 1169.482836] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 21:58:43 (1789264723) [ 1169.551958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1169.567618] LNet: 92717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1169.572096] LNetError: 92717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1169.576885] LNet: Removed LNI 192.168.202.52@tcp [ 1169.590567] LNet: Removed LNI 192.168.202.52@tcp1 [ 1169.615895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1169.644349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1169.656081] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1170.094595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1170.139586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1170.169737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1170.243236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1170.270886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1170.321740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1173.649891] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 21:58:48 (1789264728) [ 1173.835112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1173.855965] LNet: 93452:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1173.859568] LNetError: 93452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1173.862841] LNetError: 93452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1173.887475] LNet: Removed LNI 192.168.202.52@tcp [ 1173.899794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1173.926155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1173.937441] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1178.078736] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 21:58:52 (1789264732) [ 1178.186201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1178.199624] LNet: 94153:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1178.207695] LNetError: 94153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1178.221915] LNet: Removed LNI 192.168.202.52@tcp [ 1178.234769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1178.261317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1178.276093] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1182.322955] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 21:58:56 (1789264736) [ 1183.319918] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1184.295082] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 21:58:58 (1789264738) [ 1184.585892] LNet: 95188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1184.590491] LNetError: 95188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1184.601132] LNet: Removed LNI 192.168.202.52@tcp [ 1184.994990] Key type .llcrypt unregistered [ 1184.996573] Key type ._llcrypt unregistered [ 1188.244655] Key type ._llcrypt registered [ 1188.246186] Key type .llcrypt registered [ 1188.370899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1188.379573] alg: No test for adler32 (adler32-zlib) [ 1189.226906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1189.248900] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1193.708384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.152@tcp [ 1196.122753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.152@tcp [ 1196.299584] LNet: 95602:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1196.302408] LNetError: 95602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1196.314608] LNet: Removed LNI 192.168.202.52@tcp [ 1196.558479] Key type .llcrypt unregistered [ 1196.559912] Key type ._llcrypt unregistered [ 1201.906604] Key type ._llcrypt registered [ 1201.907917] Key type .llcrypt registered [ 1202.731452] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1203.531605] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1204.469390] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 21:59:18 (1789264758) [ 1205.290655] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1206.246442] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 21:59:20 (1789264760) [ 1207.352287] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg252-server [ 1208.388317] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 21:59:22 (1789264762) [ 1209.418435] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg252-server [ 1210.301186] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 21:59:24 (1789264764) [ 1211.171388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1211.218644] alg: No test for adler32 (adler32-zlib) [ 1212.153438] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 1212.283513] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1213.896282] Key type lgssc registered [ 1214.613308] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1226.918611] Key type lgssc unregistered [ 1227.097801] LNet: 98767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1227.101736] LNetError: 98767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1227.109799] LNet: Removed LNI 192.168.202.52@tcp [ 1227.534911] Key type .llcrypt unregistered [ 1227.536924] Key type ._llcrypt unregistered [ 1236.667242] Key type ._llcrypt registered [ 1236.669466] Key type .llcrypt registered [ 1237.719792] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 21:59:51 (1789264791) [ 1238.063470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1238.087693] alg: No test for adler32 (adler32-zlib) [ 1238.957632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1239.011883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1239.041325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1239.079943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1239.108466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1244.456929] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 21:59:58 (1789264798) [ 1244.575204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1244.602733] LNet: 100173:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1244.626447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1244.694383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1244.715313] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1245.471705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1245.487412] LNet: 100323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1245.490673] LNetError: 100323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1245.504037] LNet: Removed LNI 192.168.202.52@tcp [ 1245.529177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1245.628991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1245.654818] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1246.095761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1246.111127] LNetError: 100351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1246.121947] LNet: Removed LNI 192.168.202.52@tcp [ 1246.141793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1246.189256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1246.205650] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1251.012684] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 22:00:05 (1789264805) [ 1251.140343] LNet: 100941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1251.145434] LNet: 100941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1251.150339] LNetError: 100941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1251.161669] LNet: Removed LNI 192.168.202.52@tcp [ 1251.569654] Key type .llcrypt unregistered [ 1251.571691] Key type ._llcrypt unregistered [ 1255.329614] Key type ._llcrypt registered [ 1255.331766] Key type .llcrypt registered [ 1255.470881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1255.480897] alg: No test for adler32 (adler32-zlib) [ 1256.366177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1256.415695] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1256.418400] LNet: *** cfs_fail_loc=e003, val=100*** [ 1256.887429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.52@tcp [ 1256.902710] Ping reply received with mapping: [ 1256.905245] found metadata reply NI[192.168.202.52@tcp] [ 1256.907983] 34 ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1256.911392] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1256.913687] 5a 5a 5a 5a 5a 5a 5a 5a [ 1256.988323] LNet: 101335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1256.992214] LNetError: 101335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1257.001587] LNet: Removed LNI 192.168.202.52@tcp [ 1257.339403] Key type .llcrypt unregistered [ 1257.342221] Key type ._llcrypt unregistered [ 1263.860928] Key type ._llcrypt registered [ 1263.862395] Key type .llcrypt registered [ 1264.616608] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 22:00:18 (1789264818) [ 1264.888352] Key type .llcrypt unregistered [ 1264.890799] Key type ._llcrypt unregistered [ 1268.771290] Key type ._llcrypt registered [ 1268.773223] Key type .llcrypt registered [ 1268.913882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1268.925320] alg: No test for adler32 (adler32-zlib) [ 1269.855660] LNetError: 102390:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1270.081715] Key type .llcrypt unregistered [ 1270.083401] Key type ._llcrypt unregistered [ 1273.146778] Key type ._llcrypt registered [ 1273.148256] Key type .llcrypt registered [ 1273.938281] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 22:00:28 (1789264828) [ 1274.297354] Key type .llcrypt unregistered [ 1274.298930] Key type ._llcrypt unregistered [ 1277.818327] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1277.822653] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1280.195488] Key type ._llcrypt registered [ 1280.196792] Key type .llcrypt registered [ 1280.326828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1280.335409] alg: No test for adler32 (adler32-zlib) [ 1281.194193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1281.241400] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1287.460539] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 22:00:41 (1789264841) [ 1287.584714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1287.603688] LNet: 104159:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1287.607155] LNetError: 104159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1287.648292] LNet: Removed LNI 10.1.2.3@tcp [ 1287.669331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1288.185506] Key type .llcrypt unregistered [ 1288.187505] Key type ._llcrypt unregistered [ 1291.700351] Key type ._llcrypt registered [ 1291.703148] Key type .llcrypt registered [ 1291.837672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1291.844182] alg: No test for adler32 (adler32-zlib) [ 1292.701672] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1292.744301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.52@tcp [ 1292.846221] LNet: 104447:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1292.852423] LNetError: 104447:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1293.866911] LNet: Removed LNI 192.168.202.52@tcp [ 1294.258569] Key type .llcrypt unregistered [ 1294.260390] Key type ._llcrypt unregistered [ 1300.965887] Key type ._llcrypt registered [ 1300.967880] Key type .llcrypt registered [ 1301.664498] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 22:00:56 (1789264856) [ 1302.116261] Key type .llcrypt unregistered [ 1302.119350] Key type ._llcrypt unregistered [ 1305.171720] Key type ._llcrypt registered [ 1305.173412] Key type .llcrypt registered [ 1305.266198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1305.274284] alg: No test for adler32 (adler32-zlib) [ 1310.051478] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 22:01:04 (1789264864) [ 1310.187791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1310.202153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1310.242131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1310.270910] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1310.854694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1310.879297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1310.909130] LNet: Removed LNI 192.168.202.52@tcp [ 1311.061297] LNet: 106197:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1311.268557] Key type .llcrypt unregistered [ 1311.271695] Key type ._llcrypt unregistered [ 1317.857779] Key type ._llcrypt registered [ 1317.859517] Key type .llcrypt registered [ 1318.567646] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 22:01:12 (1789264872) [ 1318.771978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1318.784093] alg: No test for adler32 (adler32-zlib) [ 1319.648935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1319.688252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1319.701711] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1320.060188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.52@tcp [ 1320.086970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.52@tcp [ 1320.102693] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1320.105855] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1320.113317] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1320, 0, 1370 [ 1320.116764] LNet: Conn 0 sched [nconns, cpt]: 2, 0 [ 1320.118442] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1320.120328] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1320.122380] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1320, 0, 1370 [ 1320.127141] LNet: Conn 1 sched [nconns, cpt]: 2, 0 [ 1323.728670] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 22:01:18 (1789264878) [ 1324.002385] LNet: 107733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1324.007363] LNetError: 107733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1324.021566] LNet: Removed LNI 192.168.202.52@tcp [ 1324.359559] Key type .llcrypt unregistered [ 1324.360717] Key type ._llcrypt unregistered [ 1327.749652] Key type ._llcrypt registered [ 1327.751569] Key type .llcrypt registered [ 1327.879181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1327.894753] alg: No test for adler32 (adler32-zlib) [ 1328.746847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1328.759526] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1333.515170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.152@tcp [ 1338.952952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1338.984539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1339.000939] LNet: Added LNI 192.168.202.52@tcp1 [8/256/0/180] [ 1339.547963] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1339.639850] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1339.648517] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1339.658316] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1341.950223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1341.967074] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1348.198980] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 22:01:42 (1789264902) [ 1348.254087] LNet: 109230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1348.266354] LNetError: 109230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1348.270968] LNet: Removed LNI 192.168.202.52@tcp [ 1349.280202] LNet: Removed LNI 192.168.202.52@tcp1 [ 1349.283152] LNetError: 109230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1349.288480] LNetError: 109230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1349.307128] LNet: Removed LNI 10.1.2.3@tcp99 [ 1349.694545] Key type .llcrypt unregistered [ 1349.696516] Key type ._llcrypt unregistered [ 1353.481519] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1353.485522] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1355.820410] Key type ._llcrypt registered [ 1355.823629] Key type .llcrypt registered [ 1355.945475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1355.954539] alg: No test for adler32 (adler32-zlib) [ 1356.813372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1356.848465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1356.862670] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1357.306461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1357.318418] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1358.704630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1358.761590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1358.795522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1358.807914] LNetError: 109981:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1358.828855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1358.847778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.52@tcp [ 1359.117481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1359.159566] LNet: 110001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1359.163902] LNetError: 110001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1359.166806] LNet: Removed LNI 10.1.2.3@tcp [ 1360.237926] LNet: Removed LNI 192.168.202.52@tcp [ 1360.663479] Key type .llcrypt unregistered [ 1360.665276] Key type ._llcrypt unregistered [ 1367.209363] Key type ._llcrypt registered [ 1367.211416] Key type .llcrypt registered [ 1368.194138] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 22:02:02 (1789264922) [ 1368.449246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1368.458601] alg: No test for adler32 (adler32-zlib) [ 1369.365136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1369.422186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1369.449730] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1369.994667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.52@tcp [ 1370.070244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg252-client@tcp [ 1374.055618] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 22:02:08 (1789264928) [ 1374.134478] LNet: 111525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1374.138741] LNetError: 111525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1374.147791] LNet: Removed LNI 192.168.202.52@tcp [ 1374.499503] Key type .llcrypt unregistered [ 1374.503125] Key type ._llcrypt unregistered [ 1378.185938] Key type ._llcrypt registered [ 1378.187641] Key type .llcrypt registered [ 1378.324177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1378.336290] alg: No test for adler32 (adler32-zlib) [ 1379.222520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1379.235079] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1379.246951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1383.197239] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 22:02:17 (1789264937) [ 1384.372868] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1385.405574] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 22:02:19 (1789264939) [ 1386.444686] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1387.548765] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 22:02:21 (1789264941) [ 1387.651670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1387.665332] LNet: 112946:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1387.670636] LNetError: 112946:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1387.687462] LNet: Removed LNI 192.168.202.52@tcp [ 1387.703409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1387.745309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1387.762204] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1388.277399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1388.308521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1394.806372] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 22:02:28 (1789264948) [ 1394.944638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1394.974109] LNet: 113652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1394.986034] LNetError: 113652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1395.024616] LNet: Removed LNI 192.168.202.52@tcp [ 1395.080347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1395.176780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1395.240819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1396.180446] Key type .llcrypt unregistered [ 1396.187061] Key type ._llcrypt unregistered [ 1412.226505] Key type ._llcrypt registered [ 1412.229033] Key type .llcrypt registered [ 1414.341078] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 22:02:47 (1789264967) [ 1414.991901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1415.039076] alg: No test for adler32 (adler32-zlib) [ 1416.200185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1416.288294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1416.350742] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1417.368901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1417.469518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.52@tcp [ 1428.206679] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 22:03:01 (1789264981) [ 1428.541171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1428.593456] LNet: 115212:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1428.609050] LNetError: 115212:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1428.638238] LNet: Removed LNI 192.168.202.52@tcp [ 1428.679732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1428.727738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1428.749713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1436.985795] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 22:03:10 (1789264990) [ 1437.159822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1437.210862] LNet: 115791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1437.263794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1437.382511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1437.974315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1438.003683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1438.125663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1438.756736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1438.808946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1438.902799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1438.946220] LNet: 115826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1438.980578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1439.083137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1439.700280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1439.739899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1439.783195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1440.228591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1440.268924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1448.281814] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 22:03:21 (1789265001) [ 1448.791188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1448.819090] LNet: 116427:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1448.854815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1448.932174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1448.978174] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1449.925392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1449.942936] LNet: Added LNI 192.168.202.52@tcp2 [8/256/0/180] [ 1451.070770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1451.119829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.52@tcp [ 1451.350831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1451.372583] LNetError: 116706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1452.384177] LNet: Removed LNI 192.168.202.52@tcp [ 1452.389539] LNetError: 116706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1452.411625] LNet: Removed LNI 192.168.202.52@tcp2 [ 1452.482299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1452.592663] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1452.601158] LNet: Added LNI 192.168.202.52@tcp2 [8/256/0/180] [ 1452.610179] LNet: 116712:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1453.491766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.52@tcp [ 1453.810658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1453.854438] LNet: 116750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1453.862150] LNet: 116750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1454.880181] LNet: Removed LNI 192.168.202.52@tcp [ 1454.882160] LNetError: 116750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1454.884900] LNetError: 116750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1454.904113] LNet: Removed LNI 192.168.202.52@tcp2 [ 1454.988279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1455.080729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1455.130759] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1455.998494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1456.023533] LNet: Added LNI 192.168.202.52@tcp2 [8/256/0/180] [ 1457.166355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.52@tcp [ 1457.209746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1457.280610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.52@tcp [ 1457.620442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1458.657043] LNet: Removed LNI 192.168.202.52@tcp [ 1459.687293] LNet: Removed LNI 192.168.202.52@tcp2 [ 1459.721344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1459.807304] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1459.821975] LNet: Added LNI 192.168.202.52@tcp2 [8/256/0/180] [ 1459.831502] LNet: 117040:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1460.546611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.52@tcp [ 1460.844021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1460.898304] LNetError: 117078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1460.907357] LNetError: 117078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1460.929963] LNet: Removed LNI 192.168.202.52@tcp [ 1462.018052] LNet: Removed LNI 192.168.202.52@tcp2 [ 1462.050727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1462.124239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1462.143741] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1462.910593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1462.939750] LNet: Added LNI 192.168.202.52@tcp2 [8/256/0/180] [ 1464.144964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1464.223690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.52@tcp [ 1464.667779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1464.683858] LNet: 117357:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1464.688833] LNet: 117357:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1465.696123] LNet: Removed LNI 192.168.202.52@tcp [ 1465.729758] LNet: Removed LNI 192.168.202.52@tcp2 [ 1465.802684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1465.948617] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1465.958546] LNet: Added LNI 192.168.202.52@tcp2 [8/256/0/180] [ 1465.965382] LNet: 117363:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1466.979900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.52@tcp [ 1467.390817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1468.448865] LNet: Removed LNI 192.168.202.52@tcp [ 1468.468358] LNet: Removed LNI 192.168.202.52@tcp2 [ 1468.541995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1468.654079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1468.712681] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1469.785901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1469.811260] LNet: Added LNI 192.168.202.52@tcp2 [8/256/0/180] [ 1471.167448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.52@tcp [ 1471.214203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1471.292915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.52@tcp [ 1471.629586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1471.664572] LNetError: 117685:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1471.672213] LNetError: 117685:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1471.685364] LNet: Removed LNI 192.168.202.52@tcp [ 1472.743392] LNet: Removed LNI 192.168.202.52@tcp2 [ 1472.806711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1472.919087] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1472.933071] LNet: Added LNI 192.168.202.52@tcp2 [8/256/0/180] [ 1472.943493] LNet: 117692:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1473.887711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.52@tcp [ 1474.146609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1474.171843] LNet: Removed LNI 192.168.202.52@tcp [ 1475.239490] LNet: Removed LNI 192.168.202.52@tcp2 [ 1475.287381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1475.407558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1475.438350] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1476.484864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1476.550151] LNet: Added LNI 192.168.202.52@tcp2 [8/256/0/180] [ 1477.732405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1477.747774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1477.797898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.52@tcp [ 1478.193125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1479.270197] LNet: Removed LNI 192.168.202.52@tcp [ 1479.299900] LNet: Removed LNI 192.168.202.52@tcp2 [ 1479.401618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1479.486074] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1479.492294] LNet: Added LNI 192.168.202.52@tcp2 [8/256/0/180] [ 1479.497371] LNet: 118032:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1479.880683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.52@tcp [ 1480.275605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1481.312429] LNet: Removed LNI 192.168.202.52@tcp [ 1481.337338] LNet: Removed LNI 192.168.202.52@tcp2 [ 1481.405548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1481.532283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1481.603101] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1482.351451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1482.375785] LNet: Added LNI 192.168.202.52@tcp2 [8/256/0/180] [ 1483.388382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.52@tcp [ 1483.445543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1483.507932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1483.575409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.52@tcp [ 1483.900768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1483.910393] LNet: 118386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1483.912784] LNet: 118386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1484.960686] LNet: Removed LNI 192.168.202.52@tcp [ 1486.003993] LNet: Removed LNI 192.168.202.52@tcp2 [ 1486.092543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1486.254105] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1486.264541] LNet: Added LNI 192.168.202.52@tcp2 [8/256/0/180] [ 1487.011327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.52@tcp [ 1495.330899] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1497.185924] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 22:04:11 (1789265051) [ 1497.413066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1497.456598] LNetError: 119152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1497.464517] LNetError: 119152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 1497.474697] LNet: Removed LNI 192.168.202.52@tcp [ 1497.487058] LNet: Removed LNI 192.168.202.52@tcp2 [ 1497.524054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1497.852155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1497.887080] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1498.705806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1498.725024] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1499.762323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1499.779967] LNet: Removed LNI 10.1.2.3@tcp [ 1499.793986] LNet: Removed LNI 192.168.202.52@tcp [ 1499.832349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1499.905363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1499.951581] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1499.957705] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1500.691150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1500.723667] LNet: Removed LNI 192.168.202.52@tcp [ 1500.734632] LNet: Removed LNI 10.1.2.3@tcp [ 1507.047969] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 22:04:21 (1789265061) [ 1507.591466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1507.669268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1507.757542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1507.794540] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1508.608628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 1508.654625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.52@tcp [ 1508.715275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.52@tcp [ 1508.752478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.52@tcp [ 1508.785544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.52@tcp [ 1508.818377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.52@tcp [ 1508.841527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.52@tcp [ 1508.876989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.52@tcp [ 1508.916218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.52@tcp [ 1508.953556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.52@tcp [ 1508.999929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.52@tcp [ 1509.185243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 1515.657908] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 22:04:29 (1789265069) [ 1515.810366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1516.903471] LNet: Removed LNI 192.168.202.52@tcp [ 1516.941796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1517.120349] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1517.373196] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1517.382834] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1517.397819] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1519.908351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1519.955439] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1521.359280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1521.400051] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1523.887783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.52@tcp [ 1523.920701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1523.998822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.52@tcp -m PUT -e local_timeout -r 1 [ 1524.071548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1524.125292] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1524.128989] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1524.372464] LNetError: 120835:0:(lib-move.c:4801:lnet_parse()) 192.168.202.52@tcp, src 192.168.202.52@tcp: Dropping PUT (error -108 looking up sender) [ 1525.408825] LNet: 121293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1525.421354] LNet: 121293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1526.496211] LNet: Removed LNI 10.1.2.3@tcp [ 1527.529100] LNet: Removed LNI 192.168.202.52@tcp [ 1528.033279] Key type .llcrypt unregistered [ 1528.037928] Key type ._llcrypt unregistered [ 1539.777102] Key type ._llcrypt registered [ 1539.779151] Key type .llcrypt registered [ 1541.037747] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 22:04:54 (1789265094) [ 1541.426556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1541.441400] alg: No test for adler32 (adler32-zlib) [ 1542.417735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1542.926165] LNet: 122143:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1543.674740] Key type .llcrypt unregistered [ 1543.679030] Key type ._llcrypt unregistered [ 1550.638427] Key type ._llcrypt registered [ 1550.641403] Key type .llcrypt registered [ 1550.848934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1550.864168] alg: No test for adler32 (adler32-zlib) [ 1551.809833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1551.839214] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1559.351024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.152@tcp [ 1559.590787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1560.184640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.152@tcp [ 1560.420779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1560.444818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1560.467359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1560.506614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1560.569613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1560.601373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1560.652821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1560.678148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1560.695615] LNet: 122591:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1560.731381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1560.769631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1560.797006] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1560.830354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1560.855663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1560.885562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1561.511258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1561.746154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1561.778520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1561.813500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1561.835508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1561.879880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1561.919441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1561.969633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1562.007901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1562.034702] LNet: 122708:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1562.055709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1562.082976] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.152@tcp [ 1562.085521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1562.136608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1562.164492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1562.192989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1562.274053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1563.024715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.152@tcp [ 1563.258237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1563.281537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1563.312157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1563.336890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1563.363542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1563.392952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1563.417444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1563.441525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1563.456909] LNet: 122825:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1563.471054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1563.495362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1563.522919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1563.548347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1563.600581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1563.635617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1564.210231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1564.561971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1564.601519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1564.635117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1564.658602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1564.694311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1564.734746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1564.760973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1564.791349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1564.821450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1564.878506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1564.906441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1564.931973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1564.959475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1564.982606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1565.247214] LNet: 122989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1565.260703] LNetError: 122989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1565.283733] LNet: Removed LNI 192.168.202.52@tcp [ 1565.669971] Key type .llcrypt unregistered [ 1565.671610] Key type ._llcrypt unregistered [ 1586.080732] Key type ._llcrypt registered [ 1586.087712] Key type .llcrypt registered [ 1588.292719] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 22:05:40 (1789265140) [ 1588.967446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1588.990588] alg: No test for adler32 (adler32-zlib) [ 1589.940757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1590.533885] LNet: 123876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1591.256636] Key type .llcrypt unregistered [ 1591.260424] Key type ._llcrypt unregistered [ 1600.281608] Key type ._llcrypt registered [ 1600.283995] Key type .llcrypt registered [ 1600.490826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1600.505118] alg: No test for adler32 (adler32-zlib) [ 1601.499710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1601.557567] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1610.122700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.152@tcp [ 1618.912220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1618.954955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1618.984983] LNet: Added LNI 192.168.202.52@tcp1 [8/256/0/180] [ 1619.898651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1620.472431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.152@tcp [ 1620.701880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1620.727696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1620.759398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1620.789260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1620.822617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1620.853941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1620.904819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1620.962218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1620.981750] LNet: 124527:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1620.997441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1621.025254] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.152@tcp [ 1621.061945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1621.126505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1621.167812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1621.208656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1621.259978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1621.833641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1622.114152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1622.152211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1622.191536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1622.226591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1622.267528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1622.294896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1622.323858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1622.348823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1622.368938] LNet: 124644:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1622.397819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1622.429811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1622.470944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1622.495877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1622.524707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1622.552947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1623.132604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.152@tcp [ 1623.302438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1623.325479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1623.350854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1623.376645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1623.404475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1623.455242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1623.503659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1623.543442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1623.561908] LNet: 124761:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1623.586556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1623.631488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1623.655602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1623.679835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1623.715355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1623.752619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1624.323911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1624.579771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1624.609973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1624.639558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1624.668936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1624.713682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1624.752272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1624.785627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1624.814989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1624.837673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1624.867965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1624.890397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1624.920676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1624.947855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1624.983674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1628.600458] LNet: 124959:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1628.610927] LNetError: 124959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1628.618722] LNet: Removed LNI 192.168.202.52@tcp [ 1628.632718] LNet: Removed LNI 192.168.202.52@tcp1 [ 1628.958209] Key type .llcrypt unregistered [ 1628.959886] Key type ._llcrypt unregistered [ 1644.093705] Key type ._llcrypt registered [ 1644.095827] Key type .llcrypt registered [ 1645.651225] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 22:06:39 (1789265199) [ 1646.055643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1646.065292] alg: No test for adler32 (adler32-zlib) [ 1647.064595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1647.135875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1647.163384] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1648.152322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1648.187308] LNet: Added LNI 192.168.202.52@tcp2 [8/256/0/180] [ 1649.335454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.52@tcp [ 1649.509935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 1650.592151] LNet: Removed LNI 192.168.202.52@tcp [ 1658.214983] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 22:06:51 (1789265211) [ 1659.901855] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 1661.799869] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 22:06:55 (1789265215) [ 1670.099751] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 22:07:03 (1789265223) [ 1677.727535] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 22:07:11 (1789265231) [ 1685.075955] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 22:07:18 (1789265238) [ 1692.661399] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 22:07:26 (1789265246) [ 1700.858068] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 22:07:34 (1789265254) [ 1711.105745] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 22:07:44 (1789265264) [ 1721.669742] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 22:07:55 (1789265275) [ 1729.752914] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 22:08:03 (1789265283) [ 1737.870973] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 22:08:11 (1789265291) [ 1746.070441] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 22:08:19 (1789265299) [ 1754.455937] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 22:08:28 (1789265308) [ 1761.462758] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 22:08:35 (1789265315) [ 1768.731855] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 22:08:42 (1789265322) [ 1776.780540] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 22:08:50 (1789265330) [ 1784.671363] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 22:08:58 (1789265338) [ 1792.863817] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 22:09:06 (1789265346) [ 1802.812599] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 22:09:16 (1789265356) [ 1802.967124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1803.040279] LNet: 136652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1803.066566] LNetError: 136652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1803.114905] LNet: Removed LNI 192.168.202.52@tcp2 [ 1803.184846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1803.311963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1803.342992] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1804.528279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1804.553920] LNet: 136801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1804.558884] LNetError: 136801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1804.579128] LNet: Removed LNI 192.168.202.52@tcp [ 1804.601201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1804.666992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1804.689347] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1812.914325] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 22:09:26 (1789265366) [ 1821.561747] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 22:09:35 (1789265375) [ 1829.721447] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 22:09:43 (1789265383) [ 1838.069799] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 22:09:51 (1789265391) [ 1846.352720] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 22:10:00 (1789265400) [ 1854.624908] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 22:10:08 (1789265408) [ 1862.160631] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 22:10:15 (1789265415) [ 1870.811600] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 22:10:24 (1789265424) [ 1877.747544] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 22:10:31 (1789265431) [ 1884.832837] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 22:10:38 (1789265438) [ 1891.764440] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 22:10:45 (1789265445) [ 1900.109496] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 22:10:53 (1789265453) [ 1908.204497] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1754 sec ========== 22:11:01 (1789265461) [ 1908.576514] LNet: 144268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1908.581966] LNetError: 144268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1908.593468] LNet: Removed LNI 192.168.202.52@tcp [ 1909.453758] Key type .llcrypt unregistered [ 1909.456330] Key type ._llcrypt unregistered [ 1923.874061] Key type ._llcrypt registered [ 1923.878794] Key type .llcrypt registered [ 1924.262902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1924.273743] alg: No test for adler32 (adler32-zlib) [ 1925.371711] Lustre: Lustre: Build Version: 2.17.57_103_g3c4c45f [ 1925.632600] LNet: Added LNI 192.168.202.52@tcp [8/256/0/180] [ 1927.280296] Key type lgssc registered [ 1928.506964] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1995.726205] Lustre: Mounted lustre-client - version 2.17.57_103_g3c4c45f [ 2002.489327] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2010.673462] Lustre: Unmounted lustre-client [ 2036.883569] Key type lgssc unregistered [ 2037.121832] LNet: 147215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2037.146342] LNetError: 147215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2037.166842] LNet: Removed LNI 192.168.202.52@tcp [ 2037.903164] Key type .llcrypt unregistered [ 2037.904448] Key type ._llcrypt unregistered