[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 400368663 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2421 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22BD 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 00227D (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2331 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23C1 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE23F9 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22bd-0xbffe2330] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22bc] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2331-0xbffe23c0] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23c1-0xbffe23f8] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe23f9-0xbffe2420] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001017] APIC: Switch to symmetric I/O mode setup [ 0.002335] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007667] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008026] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010119] LSM: Security Framework initializing [ 0.011045] Yama: becoming mindful. [ 0.012038] SELinux: Initializing. [ 0.013049] *** VALIDATE selinux *** [ 0.020474] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024713] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025170] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026140] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028052] *** VALIDATE tmpfs *** [ 0.029457] *** VALIDATE proc *** [ 0.031211] *** VALIDATE cgroup *** [ 0.032008] *** VALIDATE cgroup2 *** [ 0.033256] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034157] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036029] Spectre V2 : User space: Vulnerable [ 0.037007] Speculative Store Bypass: Vulnerable [ 0.040402] debug: unmapping init [mem 0xffffffff8f259000-0xffffffff8f260fff] [ 0.043138] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044509] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045019] ... version: 2 [ 0.045874] ... bit width: 48 [ 0.046009] ... generic registers: 4 [ 0.046848] ... value mask: 0000ffffffffffff [ 0.047007] ... max period: 00007fffffffffff [ 0.048006] ... fixed-purpose events: 3 [ 0.048903] ... event mask: 000000070000000f [ 0.050239] rcu: Hierarchical SRCU implementation. [ 0.052369] smp: Bringing up secondary CPUs ... [ 0.053519] x86: Booting SMP configuration: [ 0.054016] .... node #0, CPUs: #1 #2 #3 [ 0.058010] smp: Brought up 1 node, 4 CPUs [ 0.060016] smpboot: Max logical packages: 1 [ 0.061011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.230000] node 0 deferred pages initialised in 167ms [ 0.233230] devtmpfs: initialized [ 0.234281] x86/mm: Memory block size: 128MB [ 0.237825] gcov: version magic: 0x41383552 [ 0.241491] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.246068] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.248264] pinctrl core: initialized pinctrl subsystem [ 0.249141] [ 0.249852] ************************************************************* [ 0.253019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.255011] ** ** [ 0.258013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.261028] ** ** [ 0.263010] ** This means that this kernel is built to expose internal ** [ 0.265073] ** IOMMU data structures, which may compromise security on ** [ 0.267009] ** your system. ** [ 0.269010] ** ** [ 0.271010] ** If you see this message and you are not debugging the ** [ 0.273010] ** kernel, report this immediately to your vendor! ** [ 0.275010] ** ** [ 0.277011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.279008] ************************************************************* [ 0.281810] NET: Registered protocol family 16 [ 0.283387] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.285051] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.286102] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.290016] cpuidle: using governor menu [ 0.293281] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.295893] PCI: Using configuration type 1 for base access [ 0.298134] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.307168] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.309032] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.312235] cryptd: max_cpu_qlen set to 1000 [ 0.316213] ACPI: Added _OSI(Module Device) [ 0.317014] ACPI: Added _OSI(Processor Device) [ 0.319013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.321015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.326063] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.331340] ACPI: Interpreter enabled [ 0.333063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.335012] ACPI: Using IOAPIC for interrupt routing [ 0.336112] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.339440] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.349602] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.352039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.354016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.357092] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.363327] acpiphp: Slot [2] registered [ 0.365095] acpiphp: Slot [5] registered [ 0.366093] acpiphp: Slot [6] registered [ 0.368109] acpiphp: Slot [3] registered [ 0.369071] acpiphp: Slot [4] registered [ 0.370091] acpiphp: Slot [7] registered [ 0.372099] acpiphp: Slot [8] registered [ 0.374084] acpiphp: Slot [9] registered [ 0.375069] acpiphp: Slot [10] registered [ 0.377081] acpiphp: Slot [11] registered [ 0.379080] acpiphp: Slot [12] registered [ 0.381103] acpiphp: Slot [13] registered [ 0.382119] acpiphp: Slot [14] registered [ 0.384111] acpiphp: Slot [15] registered [ 0.386088] acpiphp: Slot [16] registered [ 0.388116] acpiphp: Slot [17] registered [ 0.389140] acpiphp: Slot [18] registered [ 0.391092] acpiphp: Slot [19] registered [ 0.392100] acpiphp: Slot [20] registered [ 0.394084] acpiphp: Slot [21] registered [ 0.396094] acpiphp: Slot [22] registered [ 0.397082] acpiphp: Slot [23] registered [ 0.399104] acpiphp: Slot [24] registered [ 0.401119] acpiphp: Slot [25] registered [ 0.402187] acpiphp: Slot [26] registered [ 0.404104] acpiphp: Slot [27] registered [ 0.406098] acpiphp: Slot [28] registered [ 0.407120] acpiphp: Slot [29] registered [ 0.409097] acpiphp: Slot [30] registered [ 0.411096] acpiphp: Slot [31] registered [ 0.412059] PCI host bridge to bus 0000:00 [ 0.413013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.415037] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.418032] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.419045] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.422066] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.426038] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.428158] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.430941] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.433979] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.443531] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.448060] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.451020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.453023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.455019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.457721] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.462932] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.465039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.469307] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.475016] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.484942] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.489010] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.495204] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.500014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.503923] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.513732] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.524000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.530912] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.541019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.568018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.582403] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.586394] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.588332] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.591410] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.593192] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.598040] iommu: Default domain type: Passthrough [ 0.599299] SCSI subsystem initialized [ 0.600171] ACPI: bus type USB registered [ 0.602123] usbcore: registered new interface driver usbfs [ 0.603093] usbcore: registered new interface driver hub [ 0.604072] usbcore: registered new device driver usb [ 0.606218] pps_core: LinuxPPS API ver. 1 registered [ 0.608012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.612107] PTP clock support registered [ 0.615040] EDAC MC: Ver: 3.0.0 [ 0.616514] PCI: Using ACPI for IRQ routing [ 0.619832] NetLabel: Initializing [ 0.621008] NetLabel: domain hash size = 128 [ 0.623009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.625116] NetLabel: unlabeled traffic allowed by default [ 0.628109] vgaarb: loaded [ 0.630303] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.632009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.639069] clocksource: Switched to clocksource kvm-clock [ 0.764545] VFS: Disk quotas dquot_6.6.0 [ 0.765965] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.768778] *** VALIDATE ramfs *** [ 0.770142] *** VALIDATE hugetlbfs *** [ 0.771514] pnp: PnP ACPI init [ 0.773724] pnp: PnP ACPI: found 6 devices [ 0.788590] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.791520] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.793306] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.795254] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.797223] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.799648] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.801865] NET: Registered protocol family 2 [ 0.804051] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.809118] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.814363] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.821043] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.823533] TCP: Hash tables configured (established 65536 bind 65536) [ 0.826387] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.829438] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.831786] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.833771] NET: Registered protocol family 1 [ 0.836427] RPC: Registered named UNIX socket transport module. [ 0.840467] RPC: Registered udp transport module. [ 0.843196] RPC: Registered tcp transport module. [ 0.844917] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.847139] NET: Registered protocol family 44 [ 0.848606] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.850400] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.852215] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.854254] PCI: CLS 0 bytes, default 64 [ 0.855709] Unpacking initramfs... [ 2.260940] debug: unmapping init [mem 0xffff95517cc64000-0xffff95517ffcffff] [ 2.267490] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.271587] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.275869] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.791357] Initialise system trusted keyrings [ 2.792741] Key type blacklist registered [ 2.796084] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.814358] zbud: loaded [ 2.818153] *** VALIDATE nfs *** [ 2.819361] *** VALIDATE nfs4 *** [ 2.821474] pstore: using deflate compression [ 2.825212] Platform Keyring initialized [ 2.937958] NET: Registered protocol family 38 [ 2.940495] Key type asymmetric registered [ 2.942472] Asymmetric key parser 'x509' registered [ 2.944601] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.948221] io scheduler mq-deadline registered [ 2.949781] io scheduler kyber registered [ 2.952146] io scheduler bfq registered [ 2.953800] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.957465] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.961253] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.965588] ACPI: Power Button [PWRF] [ 2.970506] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.976672] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.013090] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.040049] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.070243] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.074646] Non-volatile memory driver v1.3 [ 3.076119] Linux agpgart interface v0.103 [ 3.104351] virtio_blk virtio1: [vda] 134136 512-byte logical blocks (68.7 MB/65.5 MiB) [ 3.106732] vda: detected capacity change from 0 to 68677632 [ 3.124526] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.126942] vdb: detected capacity change from 0 to 1073741824 [ 3.136906] libphy: Fixed MDIO Bus: probed [ 3.142712] usbcore: registered new interface driver usbserial_generic [ 3.148598] usbserial: USB Serial support registered for generic [ 3.155725] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.160422] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.162216] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.164846] mousedev: PS/2 mouse device common for all mice [ 3.168462] rtc_cmos 00:05: RTC can wake from S4 [ 3.171082] rtc_cmos 00:05: registered as rtc0 [ 3.172767] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.177526] intel_pstate: CPU model not supported [ 3.180568] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.181126] hid: raw HID events driver (C) Jiri Kosina [ 3.188612] usbcore: registered new interface driver usbhid [ 3.189431] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.191310] usbhid: USB HID core driver [ 3.191434] drop_monitor: Initializing network drop monitor service [ 3.199189] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.200830] Initializing XFRM netlink socket [ 3.207258] NET: Registered protocol family 10 [ 3.209869] Segment Routing with IPv6 [ 3.211138] NET: Registered protocol family 17 [ 3.212731] mpls_gso: MPLS GSO support [ 3.218616] RAS: Correctable Errors collector initialized. [ 3.220522] AVX version of gcm_enc/dec engaged. [ 3.221995] AES CTR mode by8 optimization enabled [ 3.328356] sched_clock: Marking stable (3328338982, 0)->(4140958637, -812619655) [ 3.331576] registered taskstats version 1 [ 3.333611] Loading compiled-in X.509 certificates [ 3.335522] zswap: loaded using pool lzo/zbud [ 3.359652] Key type big_key registered [ 3.370505] Key type encrypted registered [ 3.371686] ima: No TPM chip found, activating TPM-bypass! [ 3.373702] ima: Allocated hash algorithm: sha1 [ 3.375326] ima: No architecture policies found [ 3.377625] evm: Initialising EVM extended attributes: [ 3.380545] evm: security.selinux [ 3.381776] evm: security.ima [ 3.382990] evm: security.capability [ 3.384831] evm: HMAC attrs: 0x1 [ 3.388550] rtc_cmos 00:05: setting system clock to 2026-04-10 18:42:37 UTC (1775846557) [ 3.397319] debug: unmapping init [mem 0xffffffff90203000-0xffffffff903fffff] [ 3.400741] debug: unmapping init [mem 0xffffffff8ef82000-0xffffffff8f258fff] [ 3.409114] Write protecting the kernel read-only data: 28672k [ 3.412701] debug: unmapping init [mem 0xffffffff8d603000-0xffffffff8d7fffff] [ 3.415727] debug: unmapping init [mem 0xffffffff8df14000-0xffffffff8dffffff] [ 3.446260] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.455446] systemd[1]: Detected virtualization kvm. [ 3.457683] systemd[1]: Detected architecture x86-64. [ 3.479731] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.505541] systemd[1]: No hostname configured. [ 3.508200] systemd[1]: Set hostname to . [ 3.509900] random: systemd: uninitialized urandom read (16 bytes read) [ 3.513234] systemd[1]: Initializing machine ID from random generator. [ 3.689794] random: systemd: uninitialized urandom read (16 bytes read) [ 3.692328] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.697575] random: systemd: uninitialized urandom read (16 bytes read) [ 3.700864] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.708699] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Journal Service. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.617951] device-mapper: uevent: version 1.0.3 [ 4.621408] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.855043] random: fast init done [ 5.910965] virtio_net virtio0 ens2: renamed from eth0 [ 6.588362] scsi host0: ata_piix [ 6.603111] scsi host1: ata_piix [ 6.604321] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 6.606459] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 10.978230] random: crng init done [ 10.979588] random: 7 urandom warning(s) missed due to ratelimiting [ 13.650145] dracut-initqueue[580]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 16.368567] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.795308] printk: systemd: 23 output lines suppressed due to ratelimiting [ 19.193521] SELinux: Disabled at runtime. [ 19.279890] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 19.298164] systemd[1]: Detected virtualization kvm. [ 19.299821] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 20.186125] systemd[1]: initrd-switch-root.service: Succeeded. [ 20.190643] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 20.197590] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 20.201611] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 20.204719] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 20.213265] systemd[1]: Starting Journal Service... Starting Journal Service... [ 20.223663] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [[ 20.413458] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS  OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Paths. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ 21.116909] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /mnt. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.881944] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 22.141458] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 22.605950] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.711508] EDAC sbridge: Ver: 1.1.2 [ 25.475696] Key type dns_resolver registered [ 25.915487] NFS: Registering the id_resolver key type [ 25.919460] Key type id_resolver registered [ 25.921628] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg327-client login: [ 64.121099] libcfs: loading out-of-tree module taints kernel. [ 64.172206] Key type ._llcrypt registered [ 64.173800] Key type .llcrypt registered [ 64.499254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 64.507369] alg: No test for adler32 (adler32-zlib) [ 65.545208] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 66.010975] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 67.672164] Key type lgssc registered [ 68.409915] Lustre: Echo OBD driver; http://www.lustre.org/ [ 129.429896] Lustre: Mounted lustre-client [ 132.458551] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 145.390742] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing check_logdir /tmp/testlogs/ [ 147.863898] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing yml_node [ 150.024619] Lustre: DEBUG MARKER: Client: 2.17.52.1 [ 151.377975] Lustre: DEBUG MARKER: MDS: 2.17.52.1 [ 152.633973] Lustre: DEBUG MARKER: OSS: 2.17.52.1 [ 153.353115] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Apr 10 14:45:06 EDT 2026 [ 155.104176] Lustre: lustre-OST0000-osc-ffff9551d821f800: disconnect after 24s idle [ 160.612118] LustreError: 7900:0:(lov_obd.c:786:lov_cleanup()) lustre-clilov-ffff9551d821f800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 160.632364] Lustre: Unmounted lustre-client [ 174.983813] Key type lgssc unregistered [ 175.152717] LNet: 8345:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 175.160756] LNetError: 8345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 175.177636] LNet: Removed LNI 192.168.203.27@tcp [ 175.626204] Key type .llcrypt unregistered [ 175.627725] Key type ._llcrypt unregistered [ 189.387752] Key type ._llcrypt registered [ 189.390392] Key type .llcrypt registered [ 189.505464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 189.516198] alg: No test for adler32 (adler32-zlib) [ 190.360173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 190.398887] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 190.410166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 192.932659] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 193.028724] LNet: 9222:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 193.037177] LNetError: 9222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 193.054526] LNet: Removed LNI 192.168.203.27@tcp [ 193.465348] Key type .llcrypt unregistered [ 193.469228] Key type ._llcrypt unregistered [ 196.924605] Key type ._llcrypt registered [ 196.927904] Key type .llcrypt registered [ 197.686845] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 14:45:50 (1775846750) [ 197.857462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 197.864886] alg: No test for adler32 (adler32-zlib) [ 198.696378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 198.734733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 198.746500] LNet: 9633:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 198.758203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 198.787033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 198.799749] LNet: 9640:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 202.381994] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 14:45:55 (1775846755) [ 202.526290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 202.544812] LNet: 10217:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 202.555830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 202.600787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 206.201565] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 14:45:59 (1775846759) [ 206.244952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 206.263656] LNet: 10796:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 206.275770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 206.322824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 209.574533] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 14:46:02 (1775846762) [ 209.616800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 209.633364] LNet: 11380:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 209.661265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 209.710519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 212.792122] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 14:46:06 (1775846766) [ 212.852167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 212.872838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 212.913329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 213.100410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 216.178461] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 14:46:09 (1775846769) [ 216.244067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 216.261881] LNet: 12556:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 216.266297] LNet: 12556:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 216.293248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 216.338932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 216.542704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 216.756443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 220.109096] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 14:46:13 (1775846773) [ 220.191243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 220.220957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 220.268595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 223.510158] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 14:46:16 (1775846776) [ 223.599793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 223.625341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 223.675860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 223.704591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 223.907119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 223.925835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 224.098514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 224.122524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 224.331032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 224.352321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 224.570708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 224.591188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 224.807524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 224.822857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 225.025474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 225.056379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 225.262557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 225.284994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 225.499474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 225.526307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 225.722701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 225.750620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 225.989456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 226.015516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 226.219214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 226.251764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 226.459484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 226.483359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 229.707681] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 14:46:23 (1775846783) [ 229.745435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 229.756165] LNet: 14413:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 229.760339] LNet: 14413:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 229.775791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 229.821304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 229.841421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 232.466050] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 14:46:26 (1775846786) [ 232.498509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 232.518637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 232.567536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 232.590572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 235.328133] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 14:46:28 (1775846788) [ 235.382818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.410958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 235.446886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 235.464217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 238.159137] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 14:46:31 (1775846791) [ 238.230150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 238.252615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 238.296975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 238.314736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 241.139381] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 14:46:34 (1775846794) [ 241.182391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 241.202886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 241.243051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 241.262841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 244.053654] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 14:46:37 (1775846797) [ 244.089952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 244.113511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 244.151736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 244.178877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 246.936877] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 14:46:40 (1775846800) [ 247.009215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 247.021404] LNet: 17932:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 247.025565] LNet: 17932:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 247.045319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 247.090339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 247.278710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 250.189119] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 14:46:43 (1775846803) [ 250.239173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 250.260355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 250.348642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 250.448915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 253.120560] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 14:46:46 (1775846806) [ 253.153949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 253.171942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 253.229482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 253.326414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 255.954094] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 14:46:49 (1775846809) [ 255.997523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 256.021376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 256.239636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 256.363199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 259.315857] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 14:46:52 (1775846812) [ 259.377717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 259.403445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 259.448404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 259.472687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 262.439734] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 14:46:55 (1775846815) [ 262.507359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 262.532765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 262.572267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 262.595434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 265.662171] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 14:46:59 (1775846819) [ 265.696932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 265.722689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 265.773145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 268.768104] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 14:47:02 (1775846822) [ 268.824421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 268.846752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 268.886408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 271.675592] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 14:47:05 (1775846825) [ 271.749590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 271.769833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 271.803718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 271.990623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 274.945408] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 14:47:08 (1775846828) [ 275.000940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 275.020664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 275.062503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 275.085501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 277.780962] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 14:47:11 (1775846831) [ 277.874390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 277.897262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 277.938328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 277.959839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 280.721509] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 14:47:14 (1775846834) [ 280.764941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 280.776171] LNet: 24530:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 280.779964] LNet: 24530:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 280.796339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 280.835867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 280.854198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 283.592167] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 14:47:17 (1775846837) [ 283.622814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 283.645719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 283.684075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 283.700468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 286.624773] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 14:47:20 (1775846840) [ 286.663139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 286.690283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 286.724964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 286.754974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 286.764903] LNetError: 25721:0:(peer.c:2174:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 286.784650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 286.800351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 289.617428] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 14:47:23 (1775846843) [ 289.709978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 289.734273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 289.775905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Fdj1q6pguX [ 289.814079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Fdj1q6pguX [ 289.841846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Fdj1q6pguX [ 292.600988] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 14:47:26 (1775846846) [ 292.660570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 292.688952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 292.728241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 292.746765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 292.767813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 295.549748] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 14:47:29 (1775846849) [ 295.597321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 295.617402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 295.653597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 296.749704] LNet: Message forwarding will be enabled in 2s [ 299.808196] LNet: Message forwarding enabled [ 299.924933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 299.934843] LNet: Message forwarding will stop in 2s to 6s [ 304.125418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 304.712676] LNet: Message forwarding disabled [ 305.271102] LNet: Message forwarding will be enabled in 2s [ 307.708923] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 14:47:41 (1775846861) [ 307.740630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 308.353838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 308.390896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 310.833430] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 14:47:44 (1775846864) [ 310.893535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 310.917889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 310.945900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 310.962453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 310.977560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 310.993754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 311.010887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 311.026882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 311.041250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 311.062270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 311.079356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 311.098056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 311.113210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 311.128951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 311.143404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 311.162388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 311.183183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 311.203832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 311.220694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 311.242724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 311.261182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 311.272716] LNetError: 28788:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 314.031798] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 14:47:47 (1775846867) [ 314.073389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 314.105811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 314.144580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 317.132804] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 14:47:50 (1775846870) [ 317.948272] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg327-server [ 318.811848] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 14:47:52 (1775846872) [ 318.868344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 318.894674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 318.925877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 318.940062] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 319.827138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[28-32]@tcp [ 319.843915] LNet: 30502:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 319.872775] LNetError: 30292:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.28@tcp is being used as a gateway but routing feature is not turned on [ 319.880396] LNetError: 30292:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.28@tcp has gone from up to down [ 320.868342] LNetError: 30292:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.29@tcp is being used as a gateway but routing feature is not turned on [ 320.876775] LNetError: 30292:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 321.894918] LNetError: 30292:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.28@tcp is being used as a gateway but routing feature is not turned on [ 321.902257] LNetError: 30292:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 2 previous similar messages [ 322.683189] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 14:47:56 (1775846876) [ 322.912284] LNetError: 30503:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.31@tcp has gone from up to down [ 322.921911] LNetError: 30503:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 323.461228] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg327-server [ 323.944917] LNetError: 30292:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.29@tcp is being used as a gateway but routing feature is not turned on [ 323.953341] LNetError: 30292:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 324.208180] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 14:47:57 (1775846877) [ 324.248516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 324.960192] LNetError: 30291:0:(router.c:1262:lnet_check_routers()) Failed to discover router 192.168.203.28@tcp [ 326.305156] LNetError: 31432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 326.318316] LNet: Removed LNI 192.168.203.27@tcp [ 326.328127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 326.355515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 326.370038] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 326.702702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[28-33/2]@tcp [ 326.715651] LNet: 31555:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 326.721076] LNet: 31555:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 326.731661] LNetError: 31442:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.28@tcp has gone from up to down [ 326.736485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[28-33/2]@tcp [ 326.740744] LNetError: 31442:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 329.644826] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 14:48:03 (1775846883) [ 329.719319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 329.731210] LNetError: 32142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 329.738805] LNet: Removed LNI 192.168.203.27@tcp [ 329.751126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 329.796499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 329.822468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 329.839789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 329.873734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 329.907155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 329.932299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 329.962301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 330.003559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.022739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.064647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 330.107546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.132580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.162211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 330.196439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.220359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.257466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 330.295080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.316352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.344538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 330.382370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.401373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.429752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 330.469584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.492487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.517543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 330.551763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.570242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.601437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 330.638903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.658417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.685377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 330.724836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.749490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.786447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 333.534681] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 14:48:06 (1775846886) [ 334.296320] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg327-server [ 335.076946] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 14:48:08 (1775846888) [ 335.876184] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg327-server [ 336.619586] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 14:48:10 (1775846890) [ 336.987442] Key type .llcrypt unregistered [ 336.989474] Key type ._llcrypt unregistered [ 339.636772] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 339.678405] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 339.680592] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 339.682807] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 341.926248] Key type ._llcrypt registered [ 341.928337] Key type .llcrypt registered [ 342.008389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 342.016903] alg: No test for adler32 (adler32-zlib) [ 342.856512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 342.883311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 342.898565] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 343.253674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 343.271339] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 344.286927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 344.305571] LNet: Removed LNI 10.1.2.3@tcp [ 344.368948] LNet: 34260:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 344.372637] LNetError: 34260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 344.384435] LNet: Removed LNI 192.168.203.27@tcp [ 344.671443] Key type .llcrypt unregistered [ 344.675231] Key type ._llcrypt unregistered [ 350.235251] Key type ._llcrypt registered [ 350.240336] Key type .llcrypt registered [ 351.075511] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 14:48:24 (1775846904) [ 351.425774] Key type .llcrypt unregistered [ 351.428919] Key type ._llcrypt unregistered [ 354.436276] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 354.489165] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 354.493498] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 354.499530] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 356.904647] Key type ._llcrypt registered [ 356.908319] Key type .llcrypt registered [ 357.039443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 357.047308] alg: No test for adler32 (adler32-zlib) [ 357.888524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 357.918347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 357.942480] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 358.351604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 358.370279] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 359.785500] LNet: 35748:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 359.788987] LNetError: 35748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 359.792870] LNet: Removed LNI 192.168.203.27@tcp [ 359.799872] LNet: Removed LNI 10.1.2.3@tcp [ 360.158059] Key type .llcrypt unregistered [ 360.162196] Key type ._llcrypt unregistered [ 365.391047] Key type ._llcrypt registered [ 365.393308] Key type .llcrypt registered [ 366.117918] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 14:48:39 (1775846919) [ 366.372359] Key type .llcrypt unregistered [ 366.374237] Key type ._llcrypt unregistered [ 369.054663] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 369.099207] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 369.102541] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 369.105315] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 371.460807] Key type ._llcrypt registered [ 371.462830] Key type .llcrypt registered [ 371.621805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 371.635628] alg: No test for adler32 (adler32-zlib) [ 372.476353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 372.502993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 372.521736] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 373.467078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 373.478020] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 374.450864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 374.462966] LNet: Removed LNI 10.1.2.3@tcp [ 374.474127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 374.489530] LNet: Removed LNI 10.1.2.31@tcp [ 374.498366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 374.512069] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 374.832432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 374.843025] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 375.288841] LNet: 37667:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 375.293870] LNetError: 37667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 375.298920] LNet: Removed LNI 10.1.2.3@tcp [ 375.308989] LNet: Removed LNI 10.1.2.31@tcp [ 375.634615] Key type .llcrypt unregistered [ 375.639569] Key type ._llcrypt unregistered [ 380.505990] Key type ._llcrypt registered [ 380.507656] Key type .llcrypt registered [ 381.162690] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 14:48:54 (1775846934) [ 381.318834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 381.326933] alg: No test for adler32 (adler32-zlib) [ 382.153823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 382.183627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 382.200939] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 382.627491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 382.639263] LNet: 38620:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 382.643397] LNetError: 38620:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 382.652623] LNet: Removed LNI 192.168.203.27@tcp [ 382.663137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 382.697891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.27@tcp [ 382.712253] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 383.083681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.27@tcp [ 383.109312] LNet: Removed LNI 192.168.203.27@tcp [ 383.140808] LNet: 38755:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 383.452583] Key type .llcrypt unregistered [ 383.454900] Key type ._llcrypt unregistered [ 388.485201] Key type ._llcrypt registered [ 388.487476] Key type .llcrypt registered [ 389.133627] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 14:49:02 (1775846942) [ 390.047530] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg327-server [ 390.880319] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 14:49:04 (1775846944) [ 391.141331] Key type .llcrypt unregistered [ 391.142903] Key type ._llcrypt unregistered [ 393.734349] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 393.771376] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 393.774135] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 393.777488] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 396.106799] Key type ._llcrypt registered [ 396.108654] Key type .llcrypt registered [ 396.194167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 396.202049] alg: No test for adler32 (adler32-zlib) [ 397.029318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 397.065207] LNet: Added LNI 192.168.203.27@tcp1 [8/256/0/180] [ 397.407307] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 398.473915] LNet: 40588:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 398.477901] LNetError: 40588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 398.482642] LNet: Removed LNI 192.168.203.27@tcp1 [ 398.491662] LNet: Removed LNI 10.1.2.3@tcp2 [ 398.837450] Key type .llcrypt unregistered [ 398.839295] Key type ._llcrypt unregistered [ 403.947617] Key type ._llcrypt registered [ 403.949155] Key type .llcrypt registered [ 404.569692] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 14:49:17 (1775846957) [ 404.775757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 404.785416] alg: No test for adler32 (adler32-zlib) [ 405.631736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 405.668230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 405.692595] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 406.145019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 406.158158] LNet: 41556:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 406.162425] LNetError: 41556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 406.176297] LNet: Removed LNI 192.168.203.27@tcp [ 406.188491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 406.222368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 406.240727] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 409.371237] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 14:49:22 (1775846962) [ 409.423991] LNet: 42141:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 409.428771] LNetError: 42141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 409.440443] LNet: Removed LNI 192.168.203.27@tcp [ 409.736694] Key type .llcrypt unregistered [ 409.739811] Key type ._llcrypt unregistered [ 412.219686] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 412.290716] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 412.305181] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 412.308933] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 414.547232] Key type ._llcrypt registered [ 414.549893] Key type .llcrypt registered [ 414.627916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 414.637382] alg: No test for adler32 (adler32-zlib) [ 415.459898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 415.488653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 415.500869] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 415.845229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 415.854911] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 416.885577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 416.901510] LNet: 42888:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 416.906482] LNetError: 42888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 416.911710] LNet: Removed LNI 192.168.203.27@tcp [ 416.923502] LNet: Removed LNI 10.1.2.3@tcp [ 416.936630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.973284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 416.989408] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 416.993969] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 417.195899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.208285] LNet: Removed LNI 192.168.203.27@tcp [ 417.221959] LNet: Removed LNI 10.1.2.3@tcp [ 417.235059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 417.268416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 417.281477] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 417.283484] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 417.464338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 417.477264] LNet: Added LNI 192.168.203.27@tcp2 [8/256/0/180] [ 417.925543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.937309] LNet: 43077:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 417.940540] LNet: 43077:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 417.945327] LNetError: 43077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 417.948948] LNetError: 43077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 417.952136] LNet: Removed LNI 192.168.203.27@tcp [ 417.954197] LNet: Removed LNI 10.1.2.3@tcp [ 417.961685] LNet: Removed LNI 192.168.203.27@tcp2 [ 417.973209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 418.007457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 418.021857] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 418.024684] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 418.027643] LNet: Added LNI 192.168.203.27@tcp2 [8/256/0/180] [ 420.962429] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 14:49:34 (1775846974) [ 421.012212] LNet: 43666:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 421.016945] LNetError: 43666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 421.021633] LNetError: 43666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 421.026796] LNet: Removed LNI 192.168.203.27@tcp [ 421.030532] LNet: Removed LNI 10.1.2.3@tcp [ 421.039117] LNet: Removed LNI 192.168.203.27@tcp2 [ 421.349825] Key type .llcrypt unregistered [ 421.352208] Key type ._llcrypt unregistered [ 423.799505] Key type ._llcrypt registered [ 423.801300] Key type .llcrypt registered [ 423.886323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 423.895474] alg: No test for adler32 (adler32-zlib) [ 424.718864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 424.751716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 424.764687] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 425.142902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 425.155207] LNet: 44067:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 425.159388] LNetError: 44067:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 425.169656] LNet: Removed LNI 192.168.203.27@tcp [ 425.179461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.209123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 425.224849] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 425.433268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 425.451635] LNet: Removed LNI 192.168.203.27@tcp [ 425.461570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.492795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 425.507357] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 425.707587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 425.717722] LNet: 44127:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 425.722931] LNet: 44127:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 425.728340] LNetError: 44127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 425.732438] LNetError: 44127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 425.743428] LNet: Removed LNI 192.168.203.27@tcp [ 425.755929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.789542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 425.802760] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 425.981948] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 426.051932] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 426.056399] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 426.059492] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 428.257690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.267910] LNet: 44205:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 428.271429] LNetError: 44205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 428.278781] LNet: Removed LNI 192.168.203.27@tcp [ 428.289879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.322445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 428.335403] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 431.260745] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 14:49:44 (1775846984) [ 431.306158] LNet: 44794:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 431.310136] LNetError: 44794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 431.323693] LNet: Removed LNI 192.168.203.27@tcp [ 431.606780] Key type .llcrypt unregistered [ 431.609067] Key type ._llcrypt unregistered [ 433.888835] Key type ._llcrypt registered [ 433.890866] Key type .llcrypt registered [ 433.978389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 433.988380] alg: No test for adler32 (adler32-zlib) [ 434.809836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 434.834304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 434.846441] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 435.197749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.208536] LNet: 45199:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 435.212516] LNetError: 45199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 435.224342] LNet: Removed LNI 192.168.203.27@tcp [ 435.236353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.270811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 435.286215] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 437.718503] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 14:49:51 (1775846991) [ 437.764334] LNet: 45785:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 437.771243] LNetError: 45785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 437.778607] LNet: Removed LNI 192.168.203.27@tcp [ 438.044559] Key type .llcrypt unregistered [ 438.046428] Key type ._llcrypt unregistered [ 440.157111] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 440.237877] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 440.240145] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 440.242334] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 442.476323] Key type ._llcrypt registered [ 442.477903] Key type .llcrypt registered [ 442.548578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 442.556750] alg: No test for adler32 (adler32-zlib) [ 443.366119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 443.390716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 443.402957] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 443.766673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 443.780226] LNet: 46240:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 443.783831] LNetError: 46240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 443.797851] LNet: Removed LNI 192.168.203.27@tcp [ 443.809316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 443.842349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 443.860716] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 446.593883] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 14:50:00 (1775847000) [ 446.627709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.640625] LNet: 46832:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 446.645081] LNetError: 46832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 446.654433] LNet: Removed LNI 192.168.203.27@tcp [ 446.667779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.714844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 449.118842] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 14:50:02 (1775847002) [ 449.167829] LNet: 47414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 449.447758] Key type .llcrypt unregistered [ 449.449356] Key type ._llcrypt unregistered [ 451.769601] Key type ._llcrypt registered [ 451.771406] Key type .llcrypt registered [ 451.847286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 451.854512] alg: No test for adler32 (adler32-zlib) [ 452.664150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.690672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 452.703038] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 453.057147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.067039] LNet: 47816:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 453.071755] LNetError: 47816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 453.081427] LNet: Removed LNI 192.168.203.27@tcp [ 453.090627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.123952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 453.136578] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 455.601540] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 14:50:09 (1775847009) [ 455.643458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 455.655748] LNet: 48403:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 455.660437] LNetError: 48403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 455.670627] LNet: Removed LNI 192.168.203.27@tcp [ 455.681803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 455.710235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 458.039554] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 14:50:11 (1775847011) [ 458.657128] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 459.324728] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 14:50:12 (1775847012) [ 459.954777] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 460.639439] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 14:50:14 (1775847014) [ 461.251125] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 462.046590] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 14:50:15 (1775847015) [ 462.693903] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 463.360734] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 14:50:16 (1775847016) [ 463.968980] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 464.668340] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 14:50:18 (1775847018) [ 464.704996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.716474] LNet: 50453:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 464.727445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.761451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 464.772314] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 465.140655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.27 [ 465.151133] LNet: 50591:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 465.156194] LNet: Removed LNI 192.168.203.27@tcp [ 467.578991] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 14:50:21 (1775847021) [ 467.626344] LNet: 51155:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 467.630499] LNetError: 51155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 467.636625] LNet: Removed LNI 192.168.203.27@tcp [ 467.916638] Key type .llcrypt unregistered [ 467.918093] Key type ._llcrypt unregistered [ 470.338138] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 470.456572] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 470.461528] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 470.464267] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 472.782824] Key type ._llcrypt registered [ 472.784642] Key type .llcrypt registered [ 472.854841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 472.863358] alg: No test for adler32 (adler32-zlib) [ 473.844768] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 476.438055] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 14:50:29 (1775847029) [ 476.485181] LNet: 52059:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 476.489064] LNetError: 52059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 476.499657] LNet: Removed LNI 10.1.2.3@tcp [ 476.761504] Key type .llcrypt unregistered [ 476.763132] Key type ._llcrypt unregistered [ 479.040389] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 479.156182] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 479.159652] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 479.164192] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 481.525086] Key type ._llcrypt registered [ 481.526533] Key type .llcrypt registered [ 481.594636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 481.603502] alg: No test for adler32 (adler32-zlib) [ 482.442666] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 485.179286] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 14:50:38 (1775847038) [ 485.248951] LNet: 52954:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 485.254860] LNetError: 52954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 485.271777] LNet: Removed LNI 10.1.2.3@tcp [ 485.548639] Key type .llcrypt unregistered [ 485.552399] Key type ._llcrypt unregistered [ 487.719177] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 487.820685] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 487.822547] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 487.825695] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 490.128254] Key type ._llcrypt registered [ 490.129979] Key type .llcrypt registered [ 490.201506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 490.208963] alg: No test for adler32 (adler32-zlib) [ 491.040929] LNet: 53256:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 491.049035] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 493.717290] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 14:50:47 (1775847047) [ 493.758279] LNet: 53840:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 493.762456] LNetError: 53840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 493.773464] LNet: Removed LNI 10.1.2.3@tcp [ 494.076883] Key type .llcrypt unregistered [ 494.078920] Key type ._llcrypt unregistered [ 496.434502] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 496.552820] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 496.555405] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 496.557479] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 498.896458] Key type ._llcrypt registered [ 498.897968] Key type .llcrypt registered [ 498.975268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 498.982334] alg: No test for adler32 (adler32-zlib) [ 499.805631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 499.840657] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 503.379996] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 14:50:56 (1775847056) [ 503.576260] LNet: 55075:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 503.580899] LNetError: 55075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 503.593915] LNet: Removed LNI 10.1.2.3@tcp [ 503.855423] Key type .llcrypt unregistered [ 503.856935] Key type ._llcrypt unregistered [ 506.053993] Key type ._llcrypt registered [ 506.055297] Key type .llcrypt registered [ 506.130803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 506.138943] alg: No test for adler32 (adler32-zlib) [ 506.958740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 506.976623] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 510.304478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.127@tcp [ 510.576308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp -m GET -e local_interrupt -r 1 [ 510.595455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp -m GET -e local_interrupt -r 1 [ 510.612936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.127@tcp [ 510.625941] LNet: There was an unexpected network error while writing to 192.168.203.127: rc = -22 [ 510.723631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 510.740285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 510.757138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 510.775593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 510.792215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 510.810637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 510.827778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 510.843495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 510.853352] LNet: 55511:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 510.863674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 510.880616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 510.897522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 510.914415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 510.930590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 510.949420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 510.970946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 511.205750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp -m GET -e local_dropped -r 1 [ 511.224301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp -m GET -e local_dropped -r 1 [ 511.242392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.127@tcp [ 511.252584] LNet: There was an unexpected network error while writing to 192.168.203.127: rc = -22 [ 511.358787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 511.375381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 511.391180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 511.409665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 511.434859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 511.456541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 511.482679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 511.500463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 511.512456] LNet: 55620:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 511.521940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 511.538386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 511.554920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 511.572658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 511.594847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 511.614300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 511.637481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 511.876137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp -m GET -e local_aborted -r 1 [ 511.895591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp -m GET -e local_aborted -r 1 [ 511.913163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.127@tcp [ 512.021451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 512.040029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 512.056842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 512.074166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 512.090505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 512.096569] LNet: 1 local NIs in recovery (showing 1): 192.168.203.27@tcp [ 512.106352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 512.122799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 512.137165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 512.153297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 512.172157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 512.190201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 512.205926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 512.222655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 512.240089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 512.259880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 512.464172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp -m GET -e local_no_route -r 1 [ 512.483839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp -m GET -e local_no_route -r 1 [ 512.502256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.127@tcp [ 512.513767] LNet: There was an unexpected network error while writing to 192.168.203.127: rc = -22 [ 512.517837] LNet: Skipped 2 previous similar messages [ 512.606627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 512.623130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 512.640607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 512.656559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 512.672071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 512.688874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 512.705457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 512.722384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 512.732682] LNet: 55838:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 512.736382] LNet: 55838:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 512.746588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 512.761934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 512.778676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 512.793972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 512.809852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 512.825690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 512.842281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 513.080555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp -m GET -e local_timeout -r 1 [ 513.104155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp -m GET -e local_timeout -r 1 [ 513.122392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.127@tcp [ 513.225616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 513.241644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 513.259410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 513.275754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 513.292883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 513.308896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 513.327277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 513.346050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 513.363416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 513.380226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 513.396824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 513.412262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 513.427396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 513.443907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 513.461140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 513.690282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp -m GET -e local_error -r 1 [ 513.710490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp -m GET -e local_error -r 1 [ 513.735159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.127@tcp [ 513.847616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 513.866527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 513.888808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 513.904804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 513.921552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 513.935978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 513.955582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 513.974708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 513.993971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 514.014680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 514.033306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 514.050649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 514.067288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 514.082567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 514.099470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 514.207593] LNet: 56097:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 514.211802] LNetError: 56097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 514.226556] LNet: Removed LNI 192.168.203.27@tcp [ 514.391696] Key type .llcrypt unregistered [ 514.393245] Key type ._llcrypt unregistered [ 520.843828] Key type ._llcrypt registered [ 520.845402] Key type .llcrypt registered [ 521.466616] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 14:51:14 (1775847074) [ 521.925276] Key type .llcrypt unregistered [ 521.927598] Key type ._llcrypt unregistered [ 524.358430] Key type ._llcrypt registered [ 524.360049] Key type .llcrypt registered [ 524.449270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 524.456567] alg: No test for adler32 (adler32-zlib) [ 525.275468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 525.292203] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 528.874551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.127@tcp [ 533.565501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 533.587682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 533.601024] LNet: Added LNI 192.168.203.27@tcp1 [8/256/0/180] [ 534.270224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp -m GET -e local_interrupt -r 1 [ 534.289063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp1 -m GET -e local_interrupt -r 1 [ 534.308960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp -m GET -e local_interrupt -r 1 [ 534.331181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp1 -m GET -e local_interrupt -r 1 [ 534.352611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp1 -d 192.168.203.127@tcp -m GET -e local_interrupt -r 1 [ 534.377228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp1 -d 192.168.203.127@tcp1 -m GET -e local_interrupt -r 1 [ 534.405078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp1 -d 192.168.203.27@tcp -m GET -e local_interrupt -r 1 [ 534.430865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp1 -d 192.168.203.27@tcp1 -m GET -e local_interrupt -r 1 [ 534.451465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.127@tcp [ 534.466403] LNet: There was an unexpected network error while writing to 192.168.203.127: rc = -22 [ 534.470725] LNet: 1 local NIs in recovery (showing 1): 192.168.203.27@tcp [ 534.570584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 534.586849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 534.608667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 534.626449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 534.645037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 534.662146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 534.678591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 534.694593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 534.703965] LNet: 57560:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 534.714726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 534.729613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 534.745863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 534.763019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 534.785771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 534.810066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 534.834917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 535.084810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp -m GET -e local_dropped -r 1 [ 535.105163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp1 -m GET -e local_dropped -r 1 [ 535.130625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp -m GET -e local_dropped -r 1 [ 535.152190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp1 -m GET -e local_dropped -r 1 [ 535.172169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp1 -d 192.168.203.127@tcp -m GET -e local_dropped -r 1 [ 535.190289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp1 -d 192.168.203.127@tcp1 -m GET -e local_dropped -r 1 [ 535.213247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp1 -d 192.168.203.27@tcp -m GET -e local_dropped -r 1 [ 535.238695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp1 -d 192.168.203.27@tcp1 -m GET -e local_dropped -r 1 [ 535.265207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.127@tcp [ 535.278124] LNet: There was an unexpected network error while writing to 192.168.203.127: rc = -22 [ 535.281698] LNet: Skipped 2 previous similar messages [ 535.396981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 535.415686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 535.434902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 535.457287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 535.476541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 535.491904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 535.515973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 535.532534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 535.544861] LNet: 57682:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 535.556545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 535.575977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 535.599840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 535.621543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 535.639977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 535.662372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 535.685257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 535.957816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp -m GET -e local_aborted -r 1 [ 535.979900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp1 -m GET -e local_aborted -r 1 [ 536.007414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp -m GET -e local_aborted -r 1 [ 536.036389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp1 -m GET -e local_aborted -r 1 [ 536.060195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp1 -d 192.168.203.127@tcp -m GET -e local_aborted -r 1 [ 536.088616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp1 -d 192.168.203.127@tcp1 -m GET -e local_aborted -r 1 [ 536.112165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp1 -d 192.168.203.27@tcp -m GET -e local_aborted -r 1 [ 536.134396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp1 -d 192.168.203.27@tcp1 -m GET -e local_aborted -r 1 [ 536.159295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.127@tcp [ 536.287539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 536.308555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 536.325096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 536.342842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 536.358944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 536.374698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 536.390377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 536.409617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 536.425849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 536.442723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 536.457977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 536.474286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 536.489804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 536.508679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 536.528425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 536.749381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp -m GET -e local_no_route -r 1 [ 536.770515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp1 -m GET -e local_no_route -r 1 [ 536.795065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp -m GET -e local_no_route -r 1 [ 536.816142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp1 -m GET -e local_no_route -r 1 [ 536.835169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp1 -d 192.168.203.127@tcp -m GET -e local_no_route -r 1 [ 536.853456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp1 -d 192.168.203.127@tcp1 -m GET -e local_no_route -r 1 [ 536.872661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp1 -d 192.168.203.27@tcp -m GET -e local_no_route -r 1 [ 536.891417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp1 -d 192.168.203.27@tcp1 -m GET -e local_no_route -r 1 [ 536.910296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.127@tcp [ 536.924926] LNet: There was an unexpected network error while writing to 192.168.203.127: rc = -22 [ 536.930706] LNet: Skipped 7 previous similar messages [ 537.033374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 537.049923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 537.068627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 537.084209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 537.103328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 537.120267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 537.136662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 537.152991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 537.166292] LNet: 57924:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 537.170766] LNet: 57924:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 537.180791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 537.197695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 537.214655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 537.234225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 537.251705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 537.267264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 537.282788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 537.526854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp -m GET -e local_timeout -r 1 [ 537.547603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp1 -m GET -e local_timeout -r 1 [ 537.564699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp -m GET -e local_timeout -r 1 [ 537.581716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp1 -m GET -e local_timeout -r 1 [ 537.604491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp1 -d 192.168.203.127@tcp -m GET -e local_timeout -r 1 [ 537.624993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp1 -d 192.168.203.127@tcp1 -m GET -e local_timeout -r 1 [ 537.644774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp1 -d 192.168.203.27@tcp -m GET -e local_timeout -r 1 [ 537.667417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp1 -d 192.168.203.27@tcp1 -m GET -e local_timeout -r 1 [ 537.688863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.127@tcp [ 537.797192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 537.813274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 537.829237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 537.846167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 537.863821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 537.881395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 537.909132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 537.926225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 537.943131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 537.958802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 537.974760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 537.992718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 538.009378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 538.027607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 538.045388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 538.281098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp -m GET -e local_error -r 1 [ 538.298378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp1 -m GET -e local_error -r 1 [ 538.319517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp -m GET -e local_error -r 1 [ 538.338960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp1 -m GET -e local_error -r 1 [ 538.357299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp1 -d 192.168.203.127@tcp -m GET -e local_error -r 1 [ 538.378241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp1 -d 192.168.203.127@tcp1 -m GET -e local_error -r 1 [ 538.396499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp1 -d 192.168.203.27@tcp -m GET -e local_error -r 1 [ 538.412742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp1 -d 192.168.203.27@tcp1 -m GET -e local_error -r 1 [ 538.431151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.127@tcp [ 538.530658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 538.545718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 538.563139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 538.580535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 538.597967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 538.615395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 538.633432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 538.650516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 538.665741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 538.682443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 538.700803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 538.716238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 538.732849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 538.749568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 538.766676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 541.419992] LNet: 58242:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 541.424618] LNetError: 58242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 541.430519] LNet: Removed LNI 192.168.203.27@tcp [ 542.502507] LNet: Removed LNI 192.168.203.27@tcp1 [ 542.672618] Key type .llcrypt unregistered [ 542.675091] Key type ._llcrypt unregistered [ 549.598443] Key type ._llcrypt registered [ 549.599832] Key type .llcrypt registered [ 550.209404] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 14:51:43 (1775847103) [ 550.603361] Key type .llcrypt unregistered [ 550.604824] Key type ._llcrypt unregistered [ 552.901543] Key type ._llcrypt registered [ 552.903334] Key type .llcrypt registered [ 552.999478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 553.006066] alg: No test for adler32 (adler32-zlib) [ 553.839354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 553.857796] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 557.307806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.127@tcp [ 557.596818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp -m GET -e remote_dropped -r 1 [ 557.618166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp -m GET -e remote_dropped -r 1 [ 557.636207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.127@tcp [ 557.646026] LNet: There was an unexpected network error while writing to 192.168.203.127: rc = -22 [ 557.736331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 557.751282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 557.767605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 557.784144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 557.798977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 557.816584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 557.833786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 557.848676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 557.858741] LNet: 59503:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 557.868571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 557.884322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 557.900070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 557.920695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 557.938222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 557.956872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 557.973671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 558.192719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp -m GET -e remote_error -r 1 [ 558.212397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp -m GET -e remote_error -r 1 [ 558.231268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.127@tcp [ 558.242190] LNet: There was an unexpected network error while writing to 192.168.203.127: rc = -22 [ 558.343866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 558.364203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 558.382217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 558.400396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 558.422294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 558.440519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 558.461918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 558.479655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 558.490357] LNet: 59612:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 558.500974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 558.525393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 558.551390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 558.577098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 558.599090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 558.622897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 558.643957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 558.890177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp -m GET -e remote_timeout -r 1 [ 558.913422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp -m GET -e remote_timeout -r 1 [ 558.935107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.127@tcp [ 559.037202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 559.052298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 559.067235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 559.088129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 559.109619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 559.127220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 559.145667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 559.164571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 559.184464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 559.220754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 559.241466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 559.257407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 559.275585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 559.296626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 559.315639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 559.456260] LNet: 59762:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 559.461289] LNetError: 59762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 559.471347] LNet: Removed LNI 192.168.203.27@tcp [ 559.664965] Key type .llcrypt unregistered [ 559.667435] Key type ._llcrypt unregistered [ 566.410184] Key type ._llcrypt registered [ 566.412132] Key type .llcrypt registered [ 567.025324] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 14:52:00 (1775847120) [ 567.388425] Key type .llcrypt unregistered [ 567.390475] Key type ._llcrypt unregistered [ 569.617945] Key type ._llcrypt registered [ 569.620091] Key type .llcrypt registered [ 569.696519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 569.704162] alg: No test for adler32 (adler32-zlib) [ 570.514105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 570.525912] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 573.660953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.127@tcp [ 576.841678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 576.858864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 576.870454] LNet: Added LNI 192.168.203.27@tcp1 [8/256/0/180] [ 577.440909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp -m GET -e remote_dropped -r 1 [ 577.458428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp1 -m GET -e remote_dropped -r 1 [ 577.479720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp -m GET -e remote_dropped -r 1 [ 577.501981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp1 -m GET -e remote_dropped -r 1 [ 577.524230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp1 -d 192.168.203.127@tcp -m GET -e remote_dropped -r 1 [ 577.543946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp1 -d 192.168.203.127@tcp1 -m GET -e remote_dropped -r 1 [ 577.562268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp1 -d 192.168.203.27@tcp -m GET -e remote_dropped -r 1 [ 577.581171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp1 -d 192.168.203.27@tcp1 -m GET -e remote_dropped -r 1 [ 577.600174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.127@tcp [ 577.611282] LNet: There was an unexpected network error while writing to 192.168.203.127: rc = -22 [ 577.616533] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.127@tcp [ 577.700423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 577.716867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 577.735497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 577.750906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 577.765837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 577.781693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 577.798206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 577.816156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 577.825995] LNet: 61225:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 577.837411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 577.852600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 577.872340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 577.891648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 577.909649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 577.931753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 577.956533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 577.979337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 578.214222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp -m GET -e remote_error -r 1 [ 578.237342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp1 -m GET -e remote_error -r 1 [ 578.259473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp -m GET -e remote_error -r 1 [ 578.283280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp1 -m GET -e remote_error -r 1 [ 578.304212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp1 -d 192.168.203.127@tcp -m GET -e remote_error -r 1 [ 578.322589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp1 -d 192.168.203.127@tcp1 -m GET -e remote_error -r 1 [ 578.342980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp1 -d 192.168.203.27@tcp -m GET -e remote_error -r 1 [ 578.365510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp1 -d 192.168.203.27@tcp1 -m GET -e remote_error -r 1 [ 578.386102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.127@tcp [ 578.398484] LNet: There was an unexpected network error while writing to 192.168.203.127: rc = -22 [ 578.401356] LNet: Skipped 2 previous similar messages [ 578.494606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 578.511699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 578.527626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 578.551552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 578.570933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 578.590685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 578.607651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 578.627619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 578.641906] LNet: 61348:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 578.654543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 578.673351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 578.693906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 578.715420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 578.735987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 578.754879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 578.776702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 578.794070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 579.069680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp -m GET -e remote_timeout -r 1 [ 579.093069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp1 -m GET -e remote_timeout -r 1 [ 579.119717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp -m GET -e remote_timeout -r 1 [ 579.145639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp1 -m GET -e remote_timeout -r 1 [ 579.169255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp1 -d 192.168.203.127@tcp -m GET -e remote_timeout -r 1 [ 579.191953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp1 -d 192.168.203.127@tcp1 -m GET -e remote_timeout -r 1 [ 579.216324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp1 -d 192.168.203.27@tcp -m GET -e remote_timeout -r 1 [ 579.239237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp1 -d 192.168.203.27@tcp1 -m GET -e remote_timeout -r 1 [ 579.258447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.127@tcp [ 579.367411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 579.386126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 579.402976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 579.420052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 579.435303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 579.450323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 579.469203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 579.486930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 579.508463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 579.531837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 579.553408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 579.573442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 579.594525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 579.613090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 579.639381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 579.660599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 582.097912] LNet: 61548:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 582.103330] LNetError: 61548:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 582.107996] LNet: Removed LNI 192.168.203.27@tcp [ 582.122761] LNet: Removed LNI 192.168.203.27@tcp1 [ 582.322633] Key type .llcrypt unregistered [ 582.324103] Key type ._llcrypt unregistered [ 588.581770] Key type ._llcrypt registered [ 588.583883] Key type .llcrypt registered [ 589.179577] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 14:52:22 (1775847142) [ 589.416383] Key type .llcrypt unregistered [ 589.418119] Key type ._llcrypt unregistered [ 591.724771] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 591.766076] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 591.768936] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 591.774970] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 594.057915] Key type ._llcrypt registered [ 594.059278] Key type .llcrypt registered [ 594.127681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 594.135261] alg: No test for adler32 (adler32-zlib) [ 594.964493] LNet: 62622:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 594.971714] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 594.989823] LNet: 62636:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 594.994333] LNetError: 62636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 595.010329] LNet: Removed LNI 192.168.203.27@tcp [ 595.322694] Key type .llcrypt unregistered [ 595.326715] Key type ._llcrypt unregistered [ 597.575641] Key type ._llcrypt registered [ 597.578642] Key type .llcrypt registered [ 597.654762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 597.660762] alg: No test for adler32 (adler32-zlib) [ 598.497253] LNet: 62895:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 598.505617] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 598.508198] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 598.526136] LNet: 62909:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 598.530513] LNetError: 62909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 598.535031] LNet: Removed LNI 192.168.203.27@tcp [ 598.545849] LNet: Removed LNI 10.1.2.3@tcp1 [ 598.858211] Key type .llcrypt unregistered [ 598.860604] Key type ._llcrypt unregistered [ 601.493828] Key type ._llcrypt registered [ 601.495715] Key type .llcrypt registered [ 601.570795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 601.578263] alg: No test for adler32 (adler32-zlib) [ 602.413987] LNet: 63167:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 602.423103] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 602.425191] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 602.448158] LNet: 63181:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 602.451897] LNetError: 63181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 602.457084] LNet: Removed LNI 192.168.203.27@tcp [ 602.463493] LNet: Removed LNI 10.1.2.3@tcp [ 602.796348] Key type .llcrypt unregistered [ 602.798858] Key type ._llcrypt unregistered [ 605.201441] Key type ._llcrypt registered [ 605.202930] Key type .llcrypt registered [ 605.282016] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 605.289414] alg: No test for adler32 (adler32-zlib) [ 606.123788] LNet: 63441:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 606.134299] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 606.136608] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 606.159501] LNet: 63455:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 606.164251] LNetError: 63455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 606.167898] LNet: Removed LNI 192.168.203.27@tcp [ 606.173661] LNet: Removed LNI 10.1.2.3@tcp [ 606.490549] Key type .llcrypt unregistered [ 606.492650] Key type ._llcrypt unregistered [ 609.051951] Key type ._llcrypt registered [ 609.054035] Key type .llcrypt registered [ 609.162720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 609.172951] alg: No test for adler32 (adler32-zlib) [ 610.002456] LNet: 63718:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 610.013092] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 610.015437] LNetError: 63718:0:(config.c:1593:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 610.020041] LNetError: Error -100 starting up LNI tcp [ 610.029970] LNet: Removed LNI 192.168.203.27@tcp [ 610.409581] Key type .llcrypt unregistered [ 610.411438] Key type ._llcrypt unregistered [ 614.970147] Key type ._llcrypt registered [ 614.971761] Key type .llcrypt registered [ 615.540821] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 14:52:49 (1775847169) [ 615.916700] Key type .llcrypt unregistered [ 615.918505] Key type ._llcrypt unregistered [ 618.248963] Key type ._llcrypt registered [ 618.250689] Key type .llcrypt registered [ 618.328602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 618.335718] alg: No test for adler32 (adler32-zlib) [ 619.148217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 619.163442] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 622.338907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.127@tcp [ 622.597801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp -m GET -e network_timeout -r 1 [ 622.615957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp -m GET -e network_timeout -r 1 [ 622.636672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.127@tcp [ 622.648497] LNet: There was an unexpected network error while writing to 192.168.203.127: rc = -22 [ 622.750430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 622.765835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 622.780503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 622.795787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 622.812168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 622.827814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 622.842271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 622.857819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 622.867644] LNet: 64971:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 622.876873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 622.891950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 622.906654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 622.922125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 622.937369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 622.953493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 623.050684] LNet: 65010:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 623.053812] LNetError: 65010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 623.062424] LNet: Removed LNI 192.168.203.27@tcp [ 623.214691] Key type .llcrypt unregistered [ 623.215839] Key type ._llcrypt unregistered [ 629.705111] Key type ._llcrypt registered [ 629.706772] Key type .llcrypt registered [ 629.794551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 629.801799] alg: No test for adler32 (adler32-zlib) [ 630.617348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 630.632982] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 633.950834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.127@tcp [ 637.552565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 637.569874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 637.588415] LNet: Added LNI 192.168.203.27@tcp1 [8/256/0/180] [ 638.199202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp -m GET -e network_timeout -r 1 [ 638.219282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp1 -m GET -e network_timeout -r 1 [ 638.238031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp -m GET -e network_timeout -r 1 [ 638.257504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp1 -m GET -e network_timeout -r 1 [ 638.275198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp1 -d 192.168.203.127@tcp -m GET -e network_timeout -r 1 [ 638.293990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp1 -d 192.168.203.127@tcp1 -m GET -e network_timeout -r 1 [ 638.312751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp1 -d 192.168.203.27@tcp -m GET -e network_timeout -r 1 [ 638.334210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp1 -d 192.168.203.27@tcp1 -m GET -e network_timeout -r 1 [ 638.352504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.127@tcp [ 638.363622] LNet: There was an unexpected network error while writing to 192.168.203.127: rc = -22 [ 638.457640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 638.471617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 638.487342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 638.503403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 638.518784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 638.534528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 638.550063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 638.567229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 638.576098] LNet: 65886:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 638.585346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 638.602764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 638.619775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 638.637783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 638.656747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 638.672489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 640.988396] LNet: There was an unexpected network error while writing to 192.168.203.127: rc = -22 [ 641.109269] LNet: 65959:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 641.113265] LNetError: 65959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 641.117479] LNet: Removed LNI 192.168.203.27@tcp [ 641.123572] LNet: Removed LNI 192.168.203.27@tcp1 [ 641.289716] Key type .llcrypt unregistered [ 641.291472] Key type ._llcrypt unregistered [ 647.211158] Key type ._llcrypt registered [ 647.212478] Key type .llcrypt registered [ 647.705520] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 14:53:21 (1775847201) [ 647.847573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 647.853829] alg: No test for adler32 (adler32-zlib) [ 648.664650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.687551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 648.698554] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 648.991708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 649.003404] LNet: Added LNI 192.168.203.27@tcp1 [8/256/0/180] [ 649.359067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.27@tcp [ 649.390565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 649.412803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 649.429661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 649.449892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 649.468102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 649.486314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.27@tcp [ 649.497784] LNetError: 67057:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.27@tcp to recovery [ 649.696470] LNet: There was an unexpected network error while writing to 192.168.203.27: rc = -22 [ 649.696474] LNet: 1 local NIs in recovery (showing 1): 192.168.203.27@tcp [ 651.744333] LNet: There was an unexpected network error while writing to 192.168.203.27: rc = -22 [ 652.678594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 655.840376] LNet: There was an unexpected network error while writing to 192.168.203.27: rc = -22 [ 656.908674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 656.926483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 656.952193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 656.962377] LNet: 67175:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 656.966708] LNetError: 67175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 656.970579] LNet: Removed LNI 192.168.203.27@tcp [ 656.980339] LNet: Removed LNI 192.168.203.27@tcp1 [ 656.990355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 657.018273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 657.030675] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 657.343160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 657.353990] LNet: Added LNI 192.168.203.27@tcp1 [8/256/0/180] [ 657.705436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.27@tcp [ 657.724535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 657.751912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 657.775132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 657.793275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 657.811375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 657.829489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 657.851527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.27@tcp [ 657.864179] LNetError: 67447:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.27@tcp to recovery [ 658.016417] LNet: There was an unexpected network error while writing to 192.168.203.27: rc = -22 [ 661.061537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 664.160524] LNet: There was an unexpected network error while writing to 192.168.203.27: rc = -22 [ 664.164384] LNet: Skipped 1 previous similar message [ 668.434455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 668.456450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 668.478848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 671.738123] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 14:53:45 (1775847225) [ 671.779636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.794370] LNet: 68158:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 671.798241] LNetError: 68158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 671.801437] LNetError: 68158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 671.805537] LNet: Removed LNI 192.168.203.27@tcp [ 671.812025] LNet: Removed LNI 192.168.203.27@tcp1 [ 671.830326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.860365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 671.874868] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 672.218052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 672.229425] LNet: Added LNI 192.168.203.27@tcp1 [8/256/0/180] [ 672.642710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.27@tcp [ 672.680345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 672.705738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 672.728829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 672.749128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 672.769286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 672.786478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.27@tcp --health 0 [ 672.831986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 672.864105] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.27@tcp [ 673.894049] LNet: There was an unexpected network error while writing to 192.168.203.27: rc = -22 [ 673.898299] LNet: Skipped 1 previous similar message [ 687.328676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 687.390274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 687.405884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 687.420620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.27@tcp --health 500 [ 687.463776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 693.661768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 693.687758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 693.698498] LNet: 68656:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 693.701734] LNetError: 68656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 693.705221] LNetError: 68656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 693.708579] LNet: Removed LNI 192.168.203.27@tcp [ 693.722449] LNet: Removed LNI 192.168.203.27@tcp1 [ 693.732508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 693.755561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 693.765082] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 694.045560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 694.054893] LNet: Added LNI 192.168.203.27@tcp1 [8/256/0/180] [ 694.365330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.27@tcp [ 694.380044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 694.403536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 694.416189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 694.431801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 694.446397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 694.460207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 694.475715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.27@tcp --health 0 [ 695.776432] LNet: There was an unexpected network error while writing to 192.168.203.27: rc = -22 [ 695.779760] LNet: Skipped 3 previous similar messages [ 696.623941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 707.052451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 707.068892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 707.085391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 709.825208] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 14:54:23 (1775847263) [ 709.992101] LNet: 69724:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 709.996498] LNetError: 69724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 709.999089] LNetError: 69724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 710.002910] LNet: Removed LNI 192.168.203.27@tcp [ 710.011744] LNet: Removed LNI 192.168.203.27@tcp1 [ 710.270739] Key type .llcrypt unregistered [ 710.272459] Key type ._llcrypt unregistered [ 712.337874] Key type ._llcrypt registered [ 712.339111] Key type .llcrypt registered [ 712.409269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 712.416981] alg: No test for adler32 (adler32-zlib) [ 713.229049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 713.243509] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 716.872255] Lustre: DEBUG MARKER: Initial discovery [ 716.908566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.127@tcp [ 717.820579] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 717.858868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 717.882636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.127@tcp [ 717.894809] LNet: There was an unexpected network error while writing to 192.168.203.127: rc = -22 [ 717.905543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 717.922457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.127@tcp -r 1 -m GET -l 3 [ 717.941159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.127@tcp -r 1 -m GET -e local_error [ 717.958544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.127@tcp -r 1 -m PUT -l 6 [ 718.369537] LNet: 1 local NIs in recovery (showing 1): 192.168.203.27@tcp [ 718.506560] Lustre: DEBUG MARKER: Force oleg327-client.virtnet to discover 192.168.203.127@tcp (in background) [ 718.541610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.127@tcp [ 719.075345] Lustre: DEBUG MARKER: Force oleg327-server to discover 192.168.203.27@tcp [ 719.946817] Lustre: DEBUG MARKER: Wait for 70480 [ 721.568540] LNet: There was an unexpected network error while writing to 192.168.203.127: rc = -22 [ 722.126271] Lustre: DEBUG MARKER: Finished wait on 70480 [ 722.156831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 722.175816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 722.284690] LNet: 70933:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 722.288202] LNetError: 70933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 722.297299] LNet: Removed LNI 192.168.203.27@tcp [ 722.467400] Key type .llcrypt unregistered [ 722.469077] Key type ._llcrypt unregistered [ 728.180561] Key type ._llcrypt registered [ 728.181953] Key type .llcrypt registered [ 728.664530] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 14:54:42 (1775847282) [ 728.825574] Key type .llcrypt unregistered [ 728.827193] Key type ._llcrypt unregistered [ 730.856733] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 730.894375] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 730.896640] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 730.898781] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 733.124291] Key type ._llcrypt registered [ 733.125992] Key type .llcrypt registered [ 733.206885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 733.214067] alg: No test for adler32 (adler32-zlib) [ 734.038704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 734.065392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 734.079655] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 734.383820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 734.393304] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 737.789152] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 14:54:51 (1775847291) [ 737.822533] LNet: 73001:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 737.826084] LNetError: 73001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 737.829279] LNet: Removed LNI 192.168.203.27@tcp [ 737.837645] LNet: Removed LNI 10.1.2.3@tcp [ 738.087576] Key type .llcrypt unregistered [ 738.088910] Key type ._llcrypt unregistered [ 740.384222] Key type ._llcrypt registered [ 740.385476] Key type .llcrypt registered [ 740.462136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 740.468417] alg: No test for adler32 (adler32-zlib) [ 741.273679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 741.301696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 741.315852] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 741.605670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 741.614770] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 742.003183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.27@tcp [ 743.073121] LNetError: Unexpected error -22 connecting to 192.168.203.27@tcp at host 192.168.203.27:988 [ 744.522735] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 14:54:58 (1775847298) [ 744.565168] LNet: 74132:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 744.569622] LNetError: 74132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 745.632454] LNet: Removed LNI 192.168.203.27@tcp [ 745.634349] LNetError: 74132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 745.647978] LNet: Removed LNI 10.1.2.3@tcp [ 745.911468] Key type .llcrypt unregistered [ 745.913211] Key type ._llcrypt unregistered [ 748.181419] Key type ._llcrypt registered [ 748.182887] Key type .llcrypt registered [ 748.269550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 748.277347] alg: No test for adler32 (adler32-zlib) [ 749.088653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 749.114464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 749.126953] LNet: Added LNI 192.168.203.27@tcp1 [8/256/0/180] [ 749.409165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 749.419783] LNet: Added LNI 192.168.203.27@tcp2 [8/256/0/180] [ 749.772568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.27@tcp1 --nid 192.168.203.27@tcp2 [ 751.387787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.27@tcp1 192.168.203.27@tcp2 [ 751.397492] LNetError: 74973:0:(lib-move.c:2266:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.27@tcp1 to 192.168.203.27@tcp2 - Route table may be misconfigured [ 751.408290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.27@tcp2 192.168.203.27@tcp1 [ 753.763392] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 14:55:07 (1775847307) [ 753.801585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 753.812342] LNet: 75540:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 753.814922] LNetError: 75540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 753.817478] LNet: Removed LNI 192.168.203.27@tcp1 [ 754.853950] LNet: Removed LNI 192.168.203.27@tcp2 [ 754.865505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 754.891157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 754.903405] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 755.175631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 755.185657] LNet: Added LNI 192.168.203.27@tcp1 [8/256/0/180] [ 755.518156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.27@tcp [ 755.537335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.27@tcp 192.168.203.27@tcp [ 755.554121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.27@tcp1 192.168.203.27@tcp1 [ 755.571157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.27@tcp -d 192.168.203.27@tcp -e network_timeout [ 755.588370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.27@tcp -d 192.168.203.27@tcp1 -e network_timeout [ 755.603550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.27@tcp1 -d 192.168.203.27@tcp -e network_timeout [ 755.619318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.27@tcp1 -d 192.168.203.27@tcp1 -e network_timeout [ 755.636295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.27@tcp [ 755.646628] LNet: There was an unexpected network error while writing to 192.168.203.27: rc = -22 [ 755.655611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 755.693496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 755.732355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 755.936660] LNet: 1 local NIs in recovery (showing 1): 192.168.203.27@tcp [ 758.023864] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 14:55:11 (1775847311) [ 758.064544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 758.076864] LNet: 76401:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 758.080594] LNetError: 76401:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 758.086599] LNetError: 76401:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 758.091790] LNet: Removed LNI 192.168.203.27@tcp [ 759.142738] LNet: Removed LNI 192.168.203.27@tcp1 [ 759.151671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 759.185740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 759.294677] LNet: 76443:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 759.446382] Key type .llcrypt unregistered [ 759.447839] Key type ._llcrypt unregistered [ 763.445630] Key type ._llcrypt registered [ 763.446926] Key type .llcrypt registered [ 763.926758] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 14:55:17 (1775847317) [ 764.057576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 764.064474] alg: No test for adler32 (adler32-zlib) [ 764.878354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 765.480237] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 766.139555] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 14:55:19 (1775847319) [ 766.173722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 766.185716] LNet: 77533:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 766.196919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 766.228974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 766.242292] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 766.532402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 766.542123] LNet: Added LNI 192.168.203.27@tcp1 [8/256/0/180] [ 766.888392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.27@tcp [ 766.905468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.27@tcp1 [ 766.922219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.27@tcp1 [ 769.103825] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 14:55:22 (1775847322) [ 769.732951] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg327-server [ 770.295948] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 14:55:23 (1775847323) [ 770.953858] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg327-server [ 771.543629] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 14:55:25 (1775847325) [ 772.184258] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg327-server [ 772.774615] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 14:55:26 (1775847326) [ 773.426700] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg327-server [ 774.057425] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 14:55:27 (1775847327) [ 774.748917] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg327-server [ 775.344715] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 14:55:28 (1775847328) [ 775.989816] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg327-server [ 776.599258] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 14:55:30 (1775847330) [ 777.280577] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg327-server [ 777.848029] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 14:55:31 (1775847331) [ 778.495642] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg327-server [ 779.092629] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 14:55:32 (1775847332) [ 779.759850] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg327-server [ 780.364787] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 14:55:33 (1775847333) [ 780.404517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 780.416183] LNet: 81407:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 780.419829] LNetError: 81407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 780.423172] LNet: Removed LNI 192.168.203.27@tcp [ 781.478567] LNet: Removed LNI 192.168.203.27@tcp1 [ 781.489640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.514382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 781.525485] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 781.809407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 781.839932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.27@tcp [ 781.909574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 781.919340] LNet: 81557:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 781.922982] LNetError: 81557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 781.926602] LNetError: 81557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 781.935418] LNet: Removed LNI 192.168.203.27@tcp [ 781.944510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.971597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 781.984248] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 782.277552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.27@tcp [ 783.399844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 783.410189] LNetError: 81712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 783.419596] LNet: Removed LNI 192.168.203.27@tcp [ 783.429592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 783.454899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 783.466901] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 783.747752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 783.785487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.27@tcp [ 783.867805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 784.934480] LNet: Removed LNI 192.168.203.27@tcp [ 784.946248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 784.972811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 784.985162] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 785.296980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.27@tcp [ 785.373435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 785.381381] LNet: 82005:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 785.383951] LNet: 82005:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 786.406372] LNet: Removed LNI 192.168.203.27@tcp [ 786.415619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 786.439449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 786.450952] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 786.740234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 786.774521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.27@tcp [ 786.862206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 786.872697] LNetError: 82152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 786.875920] LNetError: 82152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 786.885490] LNet: Removed LNI 192.168.203.27@tcp [ 786.893745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 786.920410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 786.932036] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 787.249808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.27@tcp [ 787.335741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 788.390722] LNet: Removed LNI 192.168.203.27@tcp [ 788.400901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 788.427214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 788.441292] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 788.744850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 788.778524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.27@tcp [ 788.858647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 789.925682] LNet: Removed LNI 192.168.203.27@tcp [ 789.935171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 789.961338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 789.973865] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 790.247903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.27@tcp [ 790.334692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 790.344419] LNet: 82590:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 790.347577] LNet: 82590:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 791.402495] LNet: Removed LNI 192.168.203.27@tcp [ 791.411441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 791.434439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 791.446711] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 791.743378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 791.774635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.27@tcp [ 791.862781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 791.872787] LNetError: 82737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 791.876320] LNetError: 82737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 791.883575] LNet: Removed LNI 192.168.203.27@tcp [ 791.892953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 791.914661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 791.926924] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 792.206619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.27@tcp [ 792.294912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 793.318525] LNet: Removed LNI 192.168.203.27@tcp [ 793.328700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 793.358520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 793.377339] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 793.671313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 793.709750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.27@tcp [ 793.813943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 794.854695] LNet: Removed LNI 192.168.203.27@tcp [ 794.864651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 794.889739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 794.901260] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 795.199341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.27@tcp [ 796.356176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 797.414452] LNet: Removed LNI 192.168.203.27@tcp [ 797.424753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 797.451409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 797.466968] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 797.771329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 797.797108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 797.814289] LNet: Removed LNI 192.168.203.27@tcp [ 797.823437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 797.849667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 797.862262] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 798.178722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 800.691304] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 14:55:54 (1775847354) [ 800.728766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 800.738450] LNet: 84018:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 800.741487] LNet: 84018:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 800.745517] LNetError: 84018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 800.748766] LNetError: 84018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 800.762529] LNet: Removed LNI 192.168.203.27@tcp [ 800.770980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 800.796546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 800.808961] LNet: Added LNI 192.168.203.27@tcp231 [8/256/0/180] [ 801.291573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 801.307548] LNet: Removed LNI 192.168.203.27@tcp231 [ 801.316970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 801.345759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 801.358804] LNet: Added LNI 192.168.203.27@tcp231 [8/256/0/180] [ 801.362142] LNet: 84167:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 801.544702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 801.560654] LNet: Removed LNI 192.168.203.27@tcp231 [ 801.568675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 801.581152] LNet: Added LNI 192.168.203.27@tcp231 [8/256/0/180] [ 802.057882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 802.072481] LNet: Removed LNI 192.168.203.27@tcp231 [ 802.081368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 802.111384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 802.123890] LNet: Added LNI 192.168.203.27@tcp231 [8/256/0/0] [ 802.126895] LNet: 84319:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 802.302804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 802.320743] LNet: Removed LNI 192.168.203.27@tcp231 [ 802.330408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 802.345490] LNet: Added LNI 192.168.203.27@tcp231 [8/256/0/0] [ 802.803944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 802.821915] LNet: Removed LNI 192.168.203.27@tcp231 [ 802.830723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 802.860506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 802.873770] LNet: Added LNI 192.168.203.27@tcp231 [8/256/0/60] [ 803.046733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 803.064968] LNet: Removed LNI 192.168.203.27@tcp231 [ 803.072614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 803.084131] LNet: Added LNI 192.168.203.27@tcp231 [8/256/0/60] [ 803.547756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 803.563498] LNet: Removed LNI 192.168.203.27@tcp231 [ 803.572760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 803.603963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 803.617079] LNet: Added LNI 192.168.203.27@tcp231 [8/256/0/180] [ 803.620324] LNet: 84621:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 803.624362] LNet: 84621:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 803.784127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 803.799794] LNet: Removed LNI 192.168.203.27@tcp231 [ 803.807780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 803.818802] LNet: Added LNI 192.168.203.27@tcp231 [8/256/0/180] [ 806.546519] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 14:56:00 (1775847360) [ 806.583138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 806.598076] LNet: Removed LNI 192.168.203.27@tcp231 [ 806.608201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 806.639700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 806.652461] LNet: Added LNI 192.168.203.27@tcp232 [8/256/0/180] [ 806.965241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 806.982169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 806.997478] LNet: Removed LNI 192.168.203.27@tcp232 [ 807.005475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 807.016972] LNet: Added LNI 192.168.203.27@tcp232 [8/256/0/180] [ 807.331948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 807.349535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 807.364594] LNet: Removed LNI 192.168.203.27@tcp232 [ 807.373668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 807.385426] LNet: Added LNI 192.168.203.27@tcp232 [8/256/0/180] [ 807.678202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 807.695499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 807.713535] LNet: Removed LNI 192.168.203.27@tcp232 [ 807.721849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 807.735152] LNet: Added LNI 192.168.203.27@tcp232 [8/256/0/180] [ 808.050039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 808.068566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 808.088616] LNet: Removed LNI 192.168.203.27@tcp232 [ 808.097143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 810.479341] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 14:56:04 (1775847364) [ 810.888623] Key type .llcrypt unregistered [ 810.890494] Key type ._llcrypt unregistered [ 813.133232] Key type ._llcrypt registered [ 813.135173] Key type .llcrypt registered [ 813.208216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 813.214739] alg: No test for adler32 (adler32-zlib) [ 814.023995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 814.038261] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 817.060406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.127@tcp [ 819.896579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 819.913209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 819.922958] LNet: Added LNI 192.168.203.27@tcp1 [8/256/0/180] [ 820.459529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp -m GET -e local_interrupt -r 2 [ 820.475879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp1 -m GET -e local_interrupt -r 2 [ 820.492838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp -m GET -e local_interrupt -r 2 [ 820.510924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp1 -m GET -e local_interrupt -r 2 [ 820.528268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp1 -d 192.168.203.127@tcp -m GET -e local_interrupt -r 2 [ 820.546330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp1 -d 192.168.203.127@tcp1 -m GET -e local_interrupt -r 2 [ 820.563502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp1 -d 192.168.203.27@tcp -m GET -e local_interrupt -r 2 [ 820.580950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp1 -d 192.168.203.27@tcp1 -m GET -e local_interrupt -r 2 [ 820.598286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.27@tcp 192.168.203.127@tcp [ 820.607792] LNet: There was an unexpected network error while writing to 192.168.203.127: rc = -22 [ 820.610672] LNet: 1 local NIs in recovery (showing 1): 192.168.203.27@tcp [ 820.697140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 820.711662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 820.726447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 820.739810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 820.752231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 820.764279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 820.777272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 820.791510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 820.799681] LNet: 87117:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 820.810065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 820.824579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 820.840442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 820.855553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 820.870022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 820.886081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 821.082388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp -m GET -e local_dropped -r 2 [ 821.099182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp1 -m GET -e local_dropped -r 2 [ 821.114461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp -m GET -e local_dropped -r 2 [ 821.131400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp1 -m GET -e local_dropped -r 2 [ 821.148624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp1 -d 192.168.203.127@tcp -m GET -e local_dropped -r 2 [ 821.166796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp1 -d 192.168.203.127@tcp1 -m GET -e local_dropped -r 2 [ 821.185620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp1 -d 192.168.203.27@tcp -m GET -e local_dropped -r 2 [ 821.205797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp1 -d 192.168.203.27@tcp1 -m GET -e local_dropped -r 2 [ 821.224541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.27@tcp 192.168.203.127@tcp [ 821.235016] LNet: There was an unexpected network error while writing to 192.168.203.127: rc = -22 [ 821.344620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 821.358647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 821.374506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 821.389560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 821.404068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 821.417660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 821.434897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 821.453332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 821.462916] LNet: 87237:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 821.472445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 821.489328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 821.502831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 821.518366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 821.533278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 821.550288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 821.758875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp -m GET -e local_aborted -r 2 [ 821.775919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp1 -m GET -e local_aborted -r 2 [ 821.794096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp -m GET -e local_aborted -r 2 [ 821.810846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp1 -m GET -e local_aborted -r 2 [ 821.827696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp1 -d 192.168.203.127@tcp -m GET -e local_aborted -r 2 [ 821.845054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp1 -d 192.168.203.127@tcp1 -m GET -e local_aborted -r 2 [ 821.862796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp1 -d 192.168.203.27@tcp -m GET -e local_aborted -r 2 [ 821.881829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp1 -d 192.168.203.27@tcp1 -m GET -e local_aborted -r 2 [ 821.901045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.27@tcp 192.168.203.127@tcp [ 821.997465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 822.011844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 822.026616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 822.040774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 822.055571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 822.070774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 822.086845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 822.101138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 822.116473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 822.131886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 822.146403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 822.161460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 822.177130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 822.193185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 822.393344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp -m GET -e local_no_route -r 2 [ 822.412646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp1 -m GET -e local_no_route -r 2 [ 822.431867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp -m GET -e local_no_route -r 2 [ 822.450883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp1 -m GET -e local_no_route -r 2 [ 822.470435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp1 -d 192.168.203.127@tcp -m GET -e local_no_route -r 2 [ 822.488339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp1 -d 192.168.203.127@tcp1 -m GET -e local_no_route -r 2 [ 822.505439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp1 -d 192.168.203.27@tcp -m GET -e local_no_route -r 2 [ 822.523220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp1 -d 192.168.203.27@tcp1 -m GET -e local_no_route -r 2 [ 822.541265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.27@tcp 192.168.203.127@tcp [ 822.552768] LNet: There was an unexpected network error while writing to 192.168.203.127: rc = -22 [ 822.556510] LNet: Skipped 2 previous similar messages [ 822.645930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 822.660823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 822.675321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 822.691697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 822.707140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 822.723761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 822.740836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 822.755708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 822.765311] LNet: 87477:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 822.768355] LNet: 87477:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 822.777688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 822.793260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 822.808090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 822.823728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 822.837439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 822.852706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 823.067124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp -m GET -e local_timeout -r 2 [ 823.084885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp1 -m GET -e local_timeout -r 2 [ 823.102198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp -m GET -e local_timeout -r 2 [ 823.120566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp1 -m GET -e local_timeout -r 2 [ 823.139331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp1 -d 192.168.203.127@tcp -m GET -e local_timeout -r 2 [ 823.158445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp1 -d 192.168.203.127@tcp1 -m GET -e local_timeout -r 2 [ 823.173828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp1 -d 192.168.203.27@tcp -m GET -e local_timeout -r 2 [ 823.190959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp1 -d 192.168.203.27@tcp1 -m GET -e local_timeout -r 2 [ 823.209223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.27@tcp 192.168.203.127@tcp [ 823.307442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 823.322492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 823.336290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 823.349495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 823.363520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 823.377332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 823.391821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 823.407175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 823.423640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 823.438778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 823.453153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 823.465635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 823.479244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 823.493208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 823.690850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp -m GET -e remote_dropped -r 2 [ 823.708766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.127@tcp1 -m GET -e remote_dropped -r 2 [ 823.727576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp -m GET -e remote_dropped -r 2 [ 823.745826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp -d 192.168.203.27@tcp1 -m GET -e remote_dropped -r 2 [ 823.762644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp1 -d 192.168.203.127@tcp -m GET -e remote_dropped -r 2 [ 823.779463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp1 -d 192.168.203.127@tcp1 -m GET -e remote_dropped -r 2 [ 823.795343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.27@tcp1 -d 192.168.203.27@tcp -m GET -e remote_dropped -r 2 [ 823.813417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.27@tcp1 -d 192.168.203.27@tcp1 -m GET -e remote_dropped -r 2 [ 823.832685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.27@tcp 192.168.203.127@tcp [ 823.843622] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.127@tcp [ 823.918894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 823.932555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 823.946514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 823.960357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 823.974841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 823.989134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 824.004403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 824.019882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 824.035661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 824.051238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 824.066417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 824.082844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 824.098044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 824.115209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 826.301807] LNet: 87791:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 826.304772] LNetError: 87791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 826.308067] LNet: Removed LNI 192.168.203.27@tcp [ 826.318409] LNet: Removed LNI 192.168.203.27@tcp1 [ 826.468747] Key type .llcrypt unregistered [ 826.470532] Key type ._llcrypt unregistered [ 832.362977] Key type ._llcrypt registered [ 832.364750] Key type .llcrypt registered [ 832.894330] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 14:56:26 (1775847386) [ 833.040309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 833.047726] alg: No test for adler32 (adler32-zlib) [ 833.859131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 833.882611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 833.895428] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 834.251436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 834.266224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.27@tcp [ 834.276307] LNetError: 88767:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.27@tcp to recovery [ 834.310137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.27@tcp [ 834.351645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 836.608300] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 14:56:30 (1775847390) [ 836.644248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 836.654196] LNet: 89352:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 836.656884] LNetError: 89352:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 836.666727] LNet: Removed LNI 192.168.203.27@tcp [ 836.675937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 836.698127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 836.708625] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 836.982495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 836.991711] LNet: Added LNI 192.168.203.27@tcp1 [8/256/0/180] [ 837.355320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.27@tcp [ 837.364481] LNetError: 89622:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.27@tcp to recovery [ 837.728345] LNet: 1 local NIs in recovery (showing 1): 192.168.203.27@tcp [ 837.728938] LNet: There was an unexpected network error while writing to 192.168.203.27: rc = -22 [ 838.446660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.27@tcp [ 842.907691] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 14:56:36 (1775847396) [ 843.571872] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg327-server [ 844.185772] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 14:56:37 (1775847397) [ 844.853807] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg327-server [ 845.513406] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 14:56:39 (1775847399) [ 846.253136] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg327-server [ 846.938619] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 14:56:40 (1775847400) [ 846.995953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 847.006735] LNet: 91250:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 847.009775] LNetError: 91250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 847.012692] LNet: Removed LNI 192.168.203.27@tcp [ 847.019631] LNet: Removed LNI 192.168.203.27@tcp1 [ 847.030555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 847.055359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 847.066835] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 847.352656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 847.378330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 847.400500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 847.430656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 847.449249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 847.466097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 849.731456] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 14:56:43 (1775847403) [ 849.771492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 849.780063] LNet: 91972:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 849.782995] LNetError: 91972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 849.785885] LNetError: 91972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 849.801441] LNet: Removed LNI 192.168.203.27@tcp [ 849.808825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 849.829684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 849.839896] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 852.344470] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 14:56:45 (1775847405) [ 852.390421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 852.406507] LNet: 92673:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 852.413498] LNetError: 92673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 852.421754] LNet: Removed LNI 192.168.203.27@tcp [ 852.432407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 852.459851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 852.472171] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 855.076252] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 14:56:48 (1775847408) [ 855.638953] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 856.263874] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 14:56:49 (1775847409) [ 856.485122] LNet: 93711:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 856.488380] LNetError: 93711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 856.500366] LNet: Removed LNI 192.168.203.27@tcp [ 856.742569] Key type .llcrypt unregistered [ 856.744051] Key type ._llcrypt unregistered [ 858.888571] Key type ._llcrypt registered [ 858.890184] Key type .llcrypt registered [ 858.957243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 858.964693] alg: No test for adler32 (adler32-zlib) [ 859.774414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 859.787295] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 863.034879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.127@tcp [ 864.735374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.127@tcp [ 864.867940] LNet: 94126:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 864.871692] LNetError: 94126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 864.889806] LNet: Removed LNI 192.168.203.27@tcp [ 865.060235] Key type .llcrypt unregistered [ 865.062969] Key type ._llcrypt unregistered [ 869.580450] Key type ._llcrypt registered [ 869.581654] Key type .llcrypt registered [ 870.135422] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 870.752914] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 871.440836] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 14:57:04 (1775847424) [ 871.587197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 871.594730] alg: No test for adler32 (adler32-zlib) [ 872.410583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 872.447744] LNet: 95211:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 872.723347] Key type .llcrypt unregistered [ 872.725496] Key type ._llcrypt unregistered [ 875.011451] Key type ._llcrypt registered [ 875.013082] Key type .llcrypt registered [ 875.091186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 875.098261] alg: No test for adler32 (adler32-zlib) [ 875.912933] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 875.915101] LNet: 95467:0:(config.c:1245:lnet_parse_routes()) Kernel parsing of LNet routes is deprecated. Consider converting to LNet YAML configuration. [ 875.919217] LNet: 95467:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 875.929300] LNetError: 95479:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.29@tcp is being used as a gateway but routing feature is not turned on [ 875.936444] LNetError: 95479:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to o2ib through 192.168.203.29@tcp has gone from up to down [ 876.127812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 894.560151] LNetError: 95479:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to o2ib through 192.168.203.30@tcp has gone from up to down [ 894.565423] LNet: 95525:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 894.569377] LNetError: 95525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 894.579518] LNet: Removed LNI 192.168.203.27@tcp [ 894.589414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 894.616174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 894.627648] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 894.630904] LNet: 95535:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 894.635719] LNet: 95535:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 894.639533] LNetError: 95535:0:(peer.c:366:lnet_peer_ni_del_locked()) Peer NI 192.168.203.29@tcp is a gateway. Can not delete it [ 894.645868] LNetError: 95535:0:(peer.c:2166:lnet_del_peer_ni()) 192.168.203.29@tcp is a router. Can not be deleted [ 894.650868] LNet: 95535:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 897.247459] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 14:57:30 (1775847450) [ 897.696331] LNetError: 95538:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to o2ib through 192.168.203.28@tcp has gone from up to down [ 897.701551] LNetError: 95538:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 897.909948] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg327-server [ 898.563151] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 14:57:32 (1775847452) [ 899.233305] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg327-server [ 899.862749] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 14:57:33 (1775847453) [ 900.445061] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 902.144239] Key type lgssc registered [ 902.585808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 910.527673] Key type lgssc unregistered [ 910.671928] LNet: 97951:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 910.674986] LNetError: 97951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 910.684559] LNet: Removed LNI 192.168.203.27@tcp [ 910.895031] Key type .llcrypt unregistered [ 910.896290] Key type ._llcrypt unregistered [ 915.780965] Key type ._llcrypt registered [ 915.782689] Key type .llcrypt registered [ 916.271980] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 14:57:49 (1775847469) [ 916.394877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 916.401777] alg: No test for adler32 (adler32-zlib) [ 917.213750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 917.242077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 917.260667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 917.279224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 917.298147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 919.637624] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 14:57:53 (1775847473) [ 919.687818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 919.699850] LNet: 99354:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 919.715100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 919.742183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 919.755717] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 920.103649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 920.117072] LNetError: 99499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 920.128565] LNet: Removed LNI 192.168.203.27@tcp [ 920.137494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 920.166879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 920.179620] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 920.357735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 920.366679] LNet: 99527:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.369405] LNet: 99527:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 920.380363] LNet: Removed LNI 192.168.203.27@tcp [ 920.389626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 920.413261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 920.424392] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 922.761781] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 14:57:56 (1775847476) [ 922.823564] LNet: 100114:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 922.828236] LNetError: 100114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 922.833042] LNetError: 100114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 922.842520] LNet: Removed LNI 192.168.203.27@tcp [ 923.118642] Key type .llcrypt unregistered [ 923.120492] Key type ._llcrypt unregistered [ 925.336058] Key type ._llcrypt registered [ 925.337769] Key type .llcrypt registered [ 925.423573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 925.429580] alg: No test for adler32 (adler32-zlib) [ 926.237535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 926.271768] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 926.273876] LNet: *** cfs_fail_loc=e003, val=100*** [ 926.557196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.27@tcp [ 926.567357] Ping reply received with mapping: [ 926.568753] found metadata reply NI[192.168.203.27@tcp] [ 926.570522] 1b cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 926.572136] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 926.574039] 5a 5a 5a 5a 5a 5a 5a 5a [ 926.625061] LNet: 100514:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.629247] LNetError: 100514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 926.639457] LNet: Removed LNI 192.168.203.27@tcp [ 926.862796] Key type .llcrypt unregistered [ 926.864058] Key type ._llcrypt unregistered [ 930.435250] Key type ._llcrypt registered [ 930.436546] Key type .llcrypt registered [ 930.888991] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 14:58:04 (1775847484) [ 931.055501] Key type .llcrypt unregistered [ 931.056798] Key type ._llcrypt unregistered [ 933.160852] Key type ._llcrypt registered [ 933.162588] Key type .llcrypt registered [ 933.235897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 933.242621] alg: No test for adler32 (adler32-zlib) [ 934.144813] LNetError: 101557:0:(api-ni.c:2731:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 934.309657] Key type .llcrypt unregistered [ 934.311685] Key type ._llcrypt unregistered [ 936.154342] Key type ._llcrypt registered [ 936.155437] Key type .llcrypt registered [ 936.678740] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 14:58:10 (1775847490) [ 936.874451] Key type .llcrypt unregistered [ 936.875844] Key type ._llcrypt unregistered [ 938.971340] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 938.974305] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 941.168382] Key type ._llcrypt registered [ 941.169876] Key type .llcrypt registered [ 941.238187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 941.244336] alg: No test for adler32 (adler32-zlib) [ 942.054971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 942.083271] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 946.086438] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 14:58:19 (1775847499) [ 946.119611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 946.131269] LNet: 103329:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 946.135498] LNetError: 103329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 946.144534] LNet: Removed LNI 10.1.2.3@tcp [ 946.154306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 946.434631] Key type .llcrypt unregistered [ 946.436528] Key type ._llcrypt unregistered [ 948.539466] Key type ._llcrypt registered [ 948.540831] Key type .llcrypt registered [ 948.617835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 948.625355] alg: No test for adler32 (adler32-zlib) [ 949.443853] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 949.469236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.27@tcp [ 949.533444] LNet: 103623:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 949.536869] LNetError: 103623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 949.546372] LNet: Removed LNI 192.168.203.27@tcp [ 949.787482] Key type .llcrypt unregistered [ 949.788829] Key type ._llcrypt unregistered [ 954.146608] Key type ._llcrypt registered [ 954.148279] Key type .llcrypt registered [ 954.746738] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 14:58:28 (1775847508) [ 955.010483] Key type .llcrypt unregistered [ 955.012346] Key type ._llcrypt unregistered [ 957.421888] Key type ._llcrypt registered [ 957.423946] Key type .llcrypt registered [ 957.511435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 957.518699] alg: No test for adler32 (adler32-zlib) [ 960.973996] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 14:58:34 (1775847514) [ 961.017829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 961.033994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 961.060455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 961.072649] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 961.371486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 961.389762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 961.415592] LNet: Removed LNI 192.168.203.27@tcp [ 961.509599] LNet: 105364:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 961.659541] Key type .llcrypt unregistered [ 961.661292] Key type ._llcrypt unregistered [ 965.597508] Key type ._llcrypt registered [ 965.598822] Key type .llcrypt registered [ 966.086871] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 14:58:39 (1775847519) [ 966.230509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 966.237752] alg: No test for adler32 (adler32-zlib) [ 967.055359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 967.088919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 967.103709] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 967.421092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.27@tcp [ 967.439429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.27@tcp [ 967.450087] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 967.452932] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 967.454889] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 967, 0, 0 [ 967.457396] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 967.459591] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 967.462490] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 967.465143] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 967, 0, 0 [ 967.469151] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 967.470740] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 967.473118] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 967.475275] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 967, 0, 1017 [ 967.478634] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 967.480949] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 967.483878] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 967.486643] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 967, 0, 1017 [ 967.489848] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 969.936985] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 14:58:43 (1775847523) [ 970.106894] LNet: 106903:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.110269] LNetError: 106903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 970.122335] LNet: Removed LNI 192.168.203.27@tcp [ 970.373486] Key type .llcrypt unregistered [ 970.375311] Key type ._llcrypt unregistered [ 972.736970] Key type ._llcrypt registered [ 972.738741] Key type .llcrypt registered [ 972.839251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 972.849950] alg: No test for adler32 (adler32-zlib) [ 973.671881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 973.686487] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 976.904037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.127@tcp [ 979.893187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 979.911812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 979.922725] LNet: Added LNI 192.168.203.27@tcp1 [8/256/0/180] [ 980.280475] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 980.320725] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 980.322956] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 980.325152] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 982.484335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 982.495161] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 986.746228] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 14:59:00 (1775847540) [ 986.797506] LNet: 108405:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 986.800720] LNetError: 108405:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 986.803982] LNet: Removed LNI 192.168.203.27@tcp [ 987.808162] LNet: Removed LNI 192.168.203.27@tcp1 [ 987.810574] LNetError: 108405:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 987.813594] LNetError: 108405:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 987.822533] LNet: Removed LNI 10.1.2.3@tcp99 [ 988.072966] Key type .llcrypt unregistered [ 988.074764] Key type ._llcrypt unregistered [ 990.285766] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 990.290101] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 992.480497] Key type ._llcrypt registered [ 992.481949] Key type .llcrypt registered [ 992.559591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 992.565731] alg: No test for adler32 (adler32-zlib) [ 993.375404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 993.399335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 993.411191] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 993.708793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 993.718396] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 994.635645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 994.659327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 994.682519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 994.693748] LNetError: 109152:0:(peer.c:2174:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 994.711595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 994.733776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.27@tcp [ 994.903257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 994.937672] LNet: 109173:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 994.940544] LNetError: 109173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 994.943846] LNet: Removed LNI 192.168.203.27@tcp [ 994.956360] LNet: Removed LNI 10.1.2.3@tcp [ 995.211495] Key type .llcrypt unregistered [ 995.212985] Key type ._llcrypt unregistered [ 999.187759] Key type ._llcrypt registered [ 999.188896] Key type .llcrypt registered [ 999.716071] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 14:59:13 (1775847553) [ 999.853932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 999.860839] alg: No test for adler32 (adler32-zlib) [ 1000.670067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1000.694725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1000.709751] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 1001.014547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.27@tcp [ 1001.045405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg327-client@tcp [ 1003.359804] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 14:59:16 (1775847556) [ 1003.401582] LNet: 110698:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1003.404574] LNetError: 110698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1003.413486] LNet: Removed LNI 192.168.203.27@tcp [ 1003.661637] Key type .llcrypt unregistered [ 1003.663059] Key type ._llcrypt unregistered [ 1005.798103] Key type ._llcrypt registered [ 1005.799888] Key type .llcrypt registered [ 1005.878810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1005.886547] alg: No test for adler32 (adler32-zlib) [ 1006.687545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1006.697773] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 1006.712186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1008.998943] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 14:59:22 (1775847562) [ 1009.570309] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1010.219991] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 14:59:23 (1775847563) [ 1010.776293] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1011.402052] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 14:59:24 (1775847564) [ 1011.432641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1011.444486] LNet: 112116:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1011.448098] LNetError: 112116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1011.455679] LNet: Removed LNI 192.168.203.27@tcp [ 1011.465394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1011.486759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1011.495919] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 1011.774282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1011.794569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1014.267389] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 14:59:27 (1775847567) [ 1014.325984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1014.337934] LNet: 112829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1014.342856] LNetError: 112829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1014.354643] LNet: Removed LNI 192.168.203.27@tcp [ 1014.365535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1014.393252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1014.409846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1014.687600] Key type .llcrypt unregistered [ 1014.689262] Key type ._llcrypt unregistered [ 1018.775427] Key type ._llcrypt registered [ 1018.776686] Key type .llcrypt registered [ 1019.285451] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 14:59:32 (1775847572) [ 1019.428988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1019.435806] alg: No test for adler32 (adler32-zlib) [ 1020.254654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1020.279283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1020.297369] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 1020.614300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1020.642166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.27@tcp [ 1023.233924] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 14:59:36 (1775847576) [ 1023.266904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1023.279146] LNet: 114377:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1023.283444] LNetError: 114377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1023.299466] LNet: Removed LNI 192.168.203.27@tcp [ 1023.310252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1023.335509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1023.353321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1025.766649] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 14:59:39 (1775847579) [ 1025.803425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1025.814317] LNet: 114957:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1025.825312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1025.859684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1025.999683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1026.014071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1026.038130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1026.186413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1026.200942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1026.231247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1026.254860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1026.284680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1026.443191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1026.457702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1026.484358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1026.641879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1026.656685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1029.204259] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 14:59:42 (1775847582) [ 1029.260380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1029.273224] LNet: 115594:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1029.276961] LNet: 115594:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1029.289248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1029.321981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1029.342888] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 1029.648809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1029.658508] LNet: Added LNI 192.168.203.27@tcp2 [8/256/0/180] [ 1030.048465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1030.074579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.27@tcp [ 1030.171394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1030.186686] LNetError: 115875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1030.189428] LNet: Removed LNI 192.168.203.27@tcp [ 1030.196630] LNet: Removed LNI 192.168.203.27@tcp2 [ 1030.205957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1030.234489] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 1030.240189] LNet: Added LNI 192.168.203.27@tcp2 [8/256/0/180] [ 1030.242832] LNet: 115881:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1030.441605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.27@tcp [ 1030.535791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1031.584170] LNet: Removed LNI 192.168.203.27@tcp [ 1031.586457] LNetError: 115918:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1031.589699] LNetError: 115918:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1031.599460] LNet: Removed LNI 192.168.203.27@tcp2 [ 1031.608834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1031.635143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1031.647812] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 1031.960772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1031.972827] LNet: Added LNI 192.168.203.27@tcp2 [8/256/0/180] [ 1032.328499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.27@tcp [ 1032.347800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1032.377939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.27@tcp [ 1032.477032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1032.486860] LNet: 116203:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1032.490039] LNet: 116203:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1033.504134] LNet: Removed LNI 192.168.203.27@tcp [ 1034.534333] LNet: Removed LNI 192.168.203.27@tcp2 [ 1034.542628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1034.569925] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 1034.573410] LNet: Added LNI 192.168.203.27@tcp2 [8/256/0/180] [ 1034.576703] LNet: 116209:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1034.776326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.27@tcp [ 1034.855096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1034.866122] LNetError: 116246:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1034.870018] LNetError: 116246:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1034.873422] LNet: Removed LNI 192.168.203.27@tcp [ 1035.942454] LNet: Removed LNI 192.168.203.27@tcp2 [ 1035.951850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1035.973967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1035.984910] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 1036.259466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1036.268683] LNet: Added LNI 192.168.203.27@tcp2 [8/256/0/180] [ 1036.592613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1036.615158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.27@tcp [ 1036.695185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1036.704348] LNet: 116526:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1036.707016] LNet: 116526:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1037.728174] LNet: Removed LNI 192.168.203.27@tcp [ 1037.736417] LNet: Removed LNI 192.168.203.27@tcp2 [ 1037.748262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1037.776724] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 1037.779213] LNet: Added LNI 192.168.203.27@tcp2 [8/256/0/180] [ 1037.781399] LNet: 116532:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1037.959144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.27@tcp [ 1038.028941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1039.072203] LNet: Removed LNI 192.168.203.27@tcp [ 1039.073729] LNetError: 116569:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1039.076802] LNetError: 116569:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1039.086313] LNet: Removed LNI 192.168.203.27@tcp2 [ 1039.094421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1039.116914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1039.127330] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 1039.413501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1039.423805] LNet: Added LNI 192.168.203.27@tcp2 [8/256/0/180] [ 1039.777857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.27@tcp [ 1039.793349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1039.815076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.27@tcp [ 1039.908529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1040.928209] LNet: Removed LNI 192.168.203.27@tcp [ 1041.958493] LNet: Removed LNI 192.168.203.27@tcp2 [ 1041.967475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1041.993031] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 1041.996540] LNet: Added LNI 192.168.203.27@tcp2 [8/256/0/180] [ 1041.999341] LNet: 116859:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1042.183269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.27@tcp [ 1042.265049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1042.279880] LNet: Removed LNI 192.168.203.27@tcp [ 1043.302324] LNet: Removed LNI 192.168.203.27@tcp2 [ 1043.310845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1043.334413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1043.346178] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 1043.625505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1043.636196] LNet: Added LNI 192.168.203.27@tcp2 [8/256/0/180] [ 1043.968723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1043.983420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1044.006908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.27@tcp [ 1044.133476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1045.152170] LNet: Removed LNI 192.168.203.27@tcp [ 1045.159318] LNet: Removed LNI 192.168.203.27@tcp2 [ 1045.168482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1045.193628] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 1045.197516] LNet: Added LNI 192.168.203.27@tcp2 [8/256/0/180] [ 1045.374279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.27@tcp [ 1045.501524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1045.511152] LNet: 117251:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1045.514487] LNet: 117251:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1046.560151] LNet: Removed LNI 192.168.203.27@tcp [ 1046.568334] LNet: Removed LNI 192.168.203.27@tcp2 [ 1046.577565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1046.603506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1046.615169] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 1046.935568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1046.947894] LNet: Added LNI 192.168.203.27@tcp2 [8/256/0/180] [ 1047.346333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.27@tcp [ 1047.367507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1047.383758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1047.414807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.27@tcp [ 1047.584961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1047.596940] LNetError: 117552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1047.601054] LNetError: 117552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 1047.604740] LNet: Removed LNI 192.168.203.27@tcp [ 1048.614500] LNet: Removed LNI 192.168.203.27@tcp2 [ 1048.623689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1048.650560] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 1048.653507] LNet: Added LNI 192.168.203.27@tcp2 [8/256/0/180] [ 1048.656517] LNet: 117558:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1048.660569] LNet: 117558:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1048.994623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.27@tcp [ 1051.840372] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1052.492388] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 15:00:06 (1775847606) [ 1052.524224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1052.534941] LNet: Removed LNI 192.168.203.27@tcp [ 1053.606528] LNet: Removed LNI 192.168.203.27@tcp2 [ 1053.617777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1053.734649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1053.745940] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 1054.055389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1054.068177] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1054.421738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1054.430132] LNet: Removed LNI 192.168.203.27@tcp [ 1054.436485] LNet: Removed LNI 10.1.2.3@tcp [ 1054.444461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1054.472536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1054.485996] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 1054.488515] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1054.650171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1054.659393] LNet: Removed LNI 192.168.203.27@tcp [ 1054.669571] LNet: Removed LNI 10.1.2.3@tcp [ 1057.240297] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 15:00:10 (1775847610) [ 1057.286397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1057.311625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1057.364845] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1057.462146] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1057.464474] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1057.466555] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1059.582423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1059.591593] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 1059.851658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1059.860907] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1060.552199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.27@tcp [ 1060.571256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1060.597384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.27@tcp -m PUT -e local_timeout -r 1 [ 1060.620347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1060.641204] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1060.642903] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1060.719380] LNetError: 119254:0:(lib-move.c:4969:lnet_parse()) 192.168.203.27@tcp, src 192.168.203.27@tcp: Dropping PUT (error -108 looking up sender) [ 1061.728357] LNet: 119711:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1061.731028] LNet: 119711:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1062.752145] LNet: Removed LNI 192.168.203.27@tcp [ 1062.759295] LNet: Removed LNI 10.1.2.3@tcp [ 1062.894573] Key type .llcrypt unregistered [ 1062.895827] Key type ._llcrypt unregistered [ 1066.637255] Key type ._llcrypt registered [ 1066.638503] Key type .llcrypt registered [ 1067.149817] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 15:00:20 (1775847620) [ 1067.287353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1067.295260] alg: No test for adler32 (adler32-zlib) [ 1068.103794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1068.263959] LNet: 120562:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1068.533411] Key type .llcrypt unregistered [ 1068.534815] Key type ._llcrypt unregistered [ 1070.734602] Key type ._llcrypt registered [ 1070.736405] Key type .llcrypt registered [ 1070.820834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1070.829822] alg: No test for adler32 (adler32-zlib) [ 1071.659901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1071.672966] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 1074.748543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.127@tcp [ 1074.833375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1075.078383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.127@tcp [ 1075.173430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1075.187255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1075.202928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1075.219368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1075.236590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1075.254333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1075.270612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1075.288085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1075.297602] LNet: 121010:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1075.306830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1075.324272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1075.340324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1075.355701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1075.372186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1075.389923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1075.659442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1075.762952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1075.779164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1075.796847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1075.808093] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.127@tcp [ 1075.814891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1075.831106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1075.851543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1075.869197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1075.887466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1075.898835] LNet: 121127:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1075.910914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1075.934505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1075.951805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1075.969744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1075.987162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1076.003760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1076.275240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.127@tcp [ 1076.379273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1076.397659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1076.415314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1076.431273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1076.447666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1076.463794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1076.480806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1076.498540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1076.515500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1076.533639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1076.552573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1076.569406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1076.585201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1076.601250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1076.875276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1076.974254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1076.989301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1077.003595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1077.019328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1077.033532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1077.047585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1077.063376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1077.079105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1077.088201] LNet: 121361:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1077.091909] LNet: 121361:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1077.101513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1077.117636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1077.134371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1077.149733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1077.165291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1077.180952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1077.318546] LNet: 121407:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1077.323467] LNetError: 121407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1077.333409] LNet: Removed LNI 192.168.203.27@tcp [ 1077.507982] Key type .llcrypt unregistered [ 1077.510917] Key type ._llcrypt unregistered [ 1084.164434] Key type ._llcrypt registered [ 1084.166087] Key type .llcrypt registered [ 1084.772109] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 15:00:38 (1775847638) [ 1084.986685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1084.995664] alg: No test for adler32 (adler32-zlib) [ 1085.864204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1086.079801] LNet: 122293:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1086.391508] Key type .llcrypt unregistered [ 1086.393202] Key type ._llcrypt unregistered [ 1089.541601] Key type ._llcrypt registered [ 1089.543764] Key type .llcrypt registered [ 1089.681916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1089.695482] alg: No test for adler32 (adler32-zlib) [ 1090.548885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1090.566469] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 1094.576571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.127@tcp [ 1098.820253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1098.846331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1098.858889] LNet: Added LNI 192.168.203.27@tcp1 [8/256/0/180] [ 1099.450174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1099.768765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.127@tcp [ 1099.808072] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.127@tcp [ 1099.916350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1099.939782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1099.960398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1099.983598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1100.007943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1100.029919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1100.049368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1100.069395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1100.083897] LNet: 122943:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1100.102365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1100.121933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1100.145289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1100.165549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1100.185542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1100.212658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1100.578481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1100.691176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1100.710188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1100.727969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1100.747917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1100.771317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1100.794742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1100.816122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1100.843276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1100.858117] LNet: 123060:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1100.872717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1100.896913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1100.927594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1100.945386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1100.974858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1100.999933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1101.343521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.127@tcp [ 1101.464286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1101.481914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1101.503679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1101.530317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1101.557282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1101.584541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1101.609947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1101.635557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1101.666289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1101.690684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1101.715914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1101.743599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1101.769949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1101.796988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1102.117602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1102.211884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1102.229942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1102.248760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1102.269879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1102.288734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1102.303882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1102.327605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1102.357148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1102.367617] LNet: 123294:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1102.372096] LNet: 123294:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1102.385748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1102.406607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1102.424305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1102.442703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1102.465557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1102.488309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1105.006111] LNet: 123374:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1105.009397] LNetError: 123374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1105.014452] LNet: Removed LNI 192.168.203.27@tcp [ 1105.027329] LNet: Removed LNI 192.168.203.27@tcp1 [ 1105.191057] Key type .llcrypt unregistered [ 1105.192620] Key type ._llcrypt unregistered [ 1111.756194] Key type ._llcrypt registered [ 1111.757300] Key type .llcrypt registered [ 1112.321722] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 15:01:05 (1775847665) [ 1115.137628] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 15:01:08 (1775847668) [ 1118.152348] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 15:01:11 (1775847671) [ 1120.940661] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 15:01:14 (1775847674) [ 1123.912107] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 15:01:17 (1775847677) [ 1126.670724] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 15:01:20 (1775847680) [ 1129.424764] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 15:01:22 (1775847682) [ 1132.050294] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 15:01:25 (1775847685) [ 1134.734177] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 15:01:28 (1775847688) [ 1137.300937] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 15:01:30 (1775847690) [ 1140.041185] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 15:01:33 (1775847693) [ 1144.520414] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 15:01:37 (1775847697) [ 1151.680666] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 15:01:43 (1775847703) [ 1156.999142] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 15:01:50 (1775847710) [ 1162.515738] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 15:01:55 (1775847715) [ 1167.998857] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 15:02:01 (1775847721) [ 1173.566800] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 15:02:06 (1775847726) [ 1180.691695] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 15:02:13 (1775847733) [ 1181.129403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1181.152080] alg: No test for adler32 (adler32-zlib) [ 1182.029741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1182.140678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1182.187317] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 1183.254325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1183.279459] LNet: 134087:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1183.286525] LNetError: 134087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1183.305277] LNet: Removed LNI 192.168.203.27@tcp [ 1183.339460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1183.412675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1183.444770] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 1188.983454] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 15:02:22 (1775847742) [ 1193.375239] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 15:02:26 (1775847746) [ 1197.152341] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 15:02:30 (1775847750) [ 1200.895480] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 15:02:34 (1775847754) [ 1204.226088] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 15:02:37 (1775847757) [ 1207.929722] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 15:02:41 (1775847761) [ 1211.805419] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 15:02:45 (1775847765) [ 1215.424305] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 15:02:48 (1775847768) [ 1218.750461] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 15:02:52 (1775847772) [ 1222.264216] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 15:02:55 (1775847775) [ 1225.539447] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 15:02:58 (1775847778) [ 1228.923606] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 15:03:02 (1775847782) [ 1232.336661] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1078 sec ========== 15:03:05 (1775847785) [ 1232.476471] LNet: 141544:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1232.481215] LNetError: 141544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1232.493375] LNet: Removed LNI 192.168.203.27@tcp [ 1232.825044] Key type .llcrypt unregistered [ 1232.829408] Key type ._llcrypt unregistered [ 1238.834470] Key type ._llcrypt registered [ 1238.848595] Key type .llcrypt registered [ 1238.971168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1238.977357] alg: No test for adler32 (adler32-zlib) [ 1239.916743] Lustre: Lustre: Build Version: 2.17.52_1_g064474b [ 1240.065087] LNet: Added LNI 192.168.203.27@tcp [8/256/0/180] [ 1241.681099] Key type lgssc registered [ 1242.368820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1273.103069] Lustre: Mounted lustre-client [ 1275.917459] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1284.048242] LustreError: 143989:0:(lov_obd.c:786:lov_cleanup()) lustre-clilov-ffff9551d8b6a000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 1284.084163] Lustre: Unmounted lustre-client [ 1297.919307] Key type lgssc unregistered [ 1298.060338] LNet: 144470:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1298.065169] LNetError: 144470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1298.075041] LNet: Removed LNI 192.168.203.27@tcp [ 1298.386125] Key type .llcrypt unregistered [ 1298.388748] Key type ._llcrypt unregistered