[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 450379852 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002353] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.010135] LSM: Security Framework initializing [ 0.011054] Yama: becoming mindful. [ 0.012038] SELinux: Initializing. [ 0.013073] *** VALIDATE selinux *** [ 0.021008] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026026] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027140] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028126] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029123] *** VALIDATE tmpfs *** [ 0.030577] *** VALIDATE proc *** [ 0.032104] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.034284] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035161] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037029] Spectre V2 : User space: Vulnerable [ 0.038008] Speculative Store Bypass: Vulnerable [ 0.041379] debug: unmapping init [mem 0xffffffff95659000-0xffffffff95660fff] [ 0.043169] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044691] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045024] ... version: 2 [ 0.046015] ... bit width: 48 [ 0.047013] ... generic registers: 4 [ 0.048013] ... value mask: 0000ffffffffffff [ 0.049015] ... max period: 00007fffffffffff [ 0.050015] ... fixed-purpose events: 3 [ 0.051012] ... event mask: 000000070000000f [ 0.052314] rcu: Hierarchical SRCU implementation. [ 0.054402] smp: Bringing up secondary CPUs ... [ 0.055581] x86: Booting SMP configuration: [ 0.056025] .... node #0, CPUs: #1 #2 #3 [ 0.059433] smp: Brought up 1 node, 4 CPUs [ 0.061011] smpboot: Max logical packages: 1 [ 0.062020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.122278] node 0 deferred pages initialised in 58ms [ 0.125144] devtmpfs: initialized [ 0.126192] x86/mm: Memory block size: 128MB [ 0.128422] gcov: version magic: 0x41383552 [ 0.131202] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.135101] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.138302] pinctrl core: initialized pinctrl subsystem [ 0.140157] [ 0.140840] ************************************************************* [ 0.143011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.145007] ** ** [ 0.147012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.150013] ** ** [ 0.153011] ** This means that this kernel is built to expose internal ** [ 0.155010] ** IOMMU data structures, which may compromise security on ** [ 0.157013] ** your system. ** [ 0.160013] ** ** [ 0.163012] ** If you see this message and you are not debugging the ** [ 0.165010] ** kernel, report this immediately to your vendor! ** [ 0.167012] ** ** [ 0.170012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.172011] ************************************************************* [ 0.174721] NET: Registered protocol family 16 [ 0.177382] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.180052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.183060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.186425] cpuidle: using governor menu [ 0.188868] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.191384] PCI: Using configuration type 1 for base access [ 0.193116] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.202049] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.205033] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.208070] cryptd: max_cpu_qlen set to 1000 [ 0.211231] ACPI: Added _OSI(Module Device) [ 0.212011] ACPI: Added _OSI(Processor Device) [ 0.213008] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.215012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.219030] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.226507] ACPI: Interpreter enabled [ 0.229060] ACPI: PM: (supports S0 S3 S4 S5) [ 0.231012] ACPI: Using IOAPIC for interrupt routing [ 0.233096] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.236370] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.249000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.251088] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.255020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.259154] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.265454] acpiphp: Slot [2] registered [ 0.266111] acpiphp: Slot [5] registered [ 0.268106] acpiphp: Slot [6] registered [ 0.269111] acpiphp: Slot [3] registered [ 0.271079] acpiphp: Slot [4] registered [ 0.273113] acpiphp: Slot [7] registered [ 0.275112] acpiphp: Slot [8] registered [ 0.277131] acpiphp: Slot [9] registered [ 0.279102] acpiphp: Slot [10] registered [ 0.281093] acpiphp: Slot [11] registered [ 0.283106] acpiphp: Slot [12] registered [ 0.285149] acpiphp: Slot [13] registered [ 0.286106] acpiphp: Slot [14] registered [ 0.287079] acpiphp: Slot [15] registered [ 0.289077] acpiphp: Slot [16] registered [ 0.290085] acpiphp: Slot [17] registered [ 0.292091] acpiphp: Slot [18] registered [ 0.293101] acpiphp: Slot [19] registered [ 0.295093] acpiphp: Slot [20] registered [ 0.296092] acpiphp: Slot [21] registered [ 0.298114] acpiphp: Slot [22] registered [ 0.299090] acpiphp: Slot [23] registered [ 0.302078] acpiphp: Slot [24] registered [ 0.304161] acpiphp: Slot [25] registered [ 0.306126] acpiphp: Slot [26] registered [ 0.307095] acpiphp: Slot [27] registered [ 0.309100] acpiphp: Slot [28] registered [ 0.310095] acpiphp: Slot [29] registered [ 0.312102] acpiphp: Slot [30] registered [ 0.314101] acpiphp: Slot [31] registered [ 0.315047] PCI host bridge to bus 0000:00 [ 0.316014] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.318017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.321020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.323019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.325023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.329025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.330155] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.333055] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.337341] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.344991] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.349056] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.353021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.357018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.360071] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.362590] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.364777] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.367047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.370671] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.376013] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.388015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.394011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.399777] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.405013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.410015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.421872] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.432186] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.439015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.448013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.466013] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.476028] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.482437] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.485417] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.487419] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.490228] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.495070] iommu: Default domain type: Passthrough [ 0.498623] SCSI subsystem initialized [ 0.501255] ACPI: bus type USB registered [ 0.503120] usbcore: registered new interface driver usbfs [ 0.506076] usbcore: registered new interface driver hub [ 0.508097] usbcore: registered new device driver usb [ 0.510237] pps_core: LinuxPPS API ver. 1 registered [ 0.513025] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.516059] PTP clock support registered [ 0.518172] EDAC MC: Ver: 3.0.0 [ 0.520125] PCI: Using ACPI for IRQ routing [ 0.521495] NetLabel: Initializing [ 0.523015] NetLabel: domain hash size = 128 [ 0.525011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.527079] NetLabel: unlabeled traffic allowed by default [ 0.529144] vgaarb: loaded [ 0.531243] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.533013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.539000] clocksource: Switched to clocksource kvm-clock [ 0.646055] VFS: Disk quotas dquot_6.6.0 [ 0.647626] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.650264] *** VALIDATE ramfs *** [ 0.652033] *** VALIDATE hugetlbfs *** [ 0.653348] pnp: PnP ACPI init [ 0.655299] pnp: PnP ACPI: found 6 devices [ 0.683521] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.686545] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.688848] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.691613] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.694469] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.696718] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.699420] NET: Registered protocol family 2 [ 0.702265] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.707835] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.712019] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.717853] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.721831] TCP: Hash tables configured (established 65536 bind 65536) [ 0.725042] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.728626] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.731786] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.735109] NET: Registered protocol family 1 [ 0.738243] RPC: Registered named UNIX socket transport module. [ 0.740662] RPC: Registered udp transport module. [ 0.743070] RPC: Registered tcp transport module. [ 0.746402] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.748949] NET: Registered protocol family 44 [ 0.750928] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.753237] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.755054] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.757641] PCI: CLS 0 bytes, default 64 [ 0.759549] Unpacking initramfs... [ 2.159729] debug: unmapping init [mem 0xffff900bfcc64000-0xffff900bfffcffff] [ 2.162302] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.164458] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.167846] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.660259] Initialise system trusted keyrings [ 2.662319] Key type blacklist registered [ 2.664592] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.674147] zbud: loaded [ 2.677478] *** VALIDATE nfs *** [ 2.679088] *** VALIDATE nfs4 *** [ 2.680711] pstore: using deflate compression [ 2.685549] Platform Keyring initialized [ 2.789769] NET: Registered protocol family 38 [ 2.791617] Key type asymmetric registered [ 2.793112] Asymmetric key parser 'x509' registered [ 2.794921] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.796978] io scheduler mq-deadline registered [ 2.799084] io scheduler kyber registered [ 2.801075] io scheduler bfq registered [ 2.803360] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.806914] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.810257] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.813658] ACPI: Power Button [PWRF] [ 2.819405] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.825490] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.840243] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.868324] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.897628] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.903373] Non-volatile memory driver v1.3 [ 2.905223] Linux agpgart interface v0.103 [ 2.939790] virtio_blk virtio1: [vda] 149784 512-byte logical blocks (76.7 MB/73.1 MiB) [ 2.943438] vda: detected capacity change from 0 to 76689408 [ 2.958976] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.962603] vdb: detected capacity change from 0 to 1073741824 [ 2.969211] libphy: Fixed MDIO Bus: probed [ 2.984722] usbcore: registered new interface driver usbserial_generic [ 2.990101] usbserial: USB Serial support registered for generic [ 2.994140] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.999555] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.001864] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.005038] mousedev: PS/2 mouse device common for all mice [ 3.011329] rtc_cmos 00:05: RTC can wake from S4 [ 3.015881] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.017843] rtc_cmos 00:05: registered as rtc0 [ 3.026936] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.027418] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.030531] intel_pstate: CPU model not supported [ 3.036718] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.042677] hid: raw HID events driver (C) Jiri Kosina [ 3.044872] usbcore: registered new interface driver usbhid [ 3.047176] usbhid: USB HID core driver [ 3.049147] drop_monitor: Initializing network drop monitor service [ 3.052148] Initializing XFRM netlink socket [ 3.054516] NET: Registered protocol family 10 [ 3.058277] Segment Routing with IPv6 [ 3.059947] NET: Registered protocol family 17 [ 3.062645] mpls_gso: MPLS GSO support [ 3.070375] RAS: Correctable Errors collector initialized. [ 3.073617] AVX version of gcm_enc/dec engaged. [ 3.076252] AES CTR mode by8 optimization enabled [ 3.178923] sched_clock: Marking stable (3178903427, 0)->(4090013786, -911110359) [ 3.183251] registered taskstats version 1 [ 3.186152] Loading compiled-in X.509 certificates [ 3.188490] zswap: loaded using pool lzo/zbud [ 3.216499] Key type big_key registered [ 3.229551] Key type encrypted registered [ 3.231414] ima: No TPM chip found, activating TPM-bypass! [ 3.233646] ima: Allocated hash algorithm: sha1 [ 3.235396] ima: No architecture policies found [ 3.237240] evm: Initialising EVM extended attributes: [ 3.239283] evm: security.selinux [ 3.240681] evm: security.ima [ 3.241755] evm: security.capability [ 3.243166] evm: HMAC attrs: 0x1 [ 3.245402] rtc_cmos 00:05: setting system clock to 2026-09-05 05:58:46 UTC (1788587926) [ 3.252517] debug: unmapping init [mem 0xffffffff96603000-0xffffffff967fffff] [ 3.256052] debug: unmapping init [mem 0xffffffff95382000-0xffffffff95658fff] [ 3.265064] Write protecting the kernel read-only data: 28672k [ 3.268947] debug: unmapping init [mem 0xffffffff93a03000-0xffffffff93bfffff] [ 3.272141] debug: unmapping init [mem 0xffffffff94314000-0xffffffff943fffff] [ 3.301106] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.310114] systemd[1]: Detected virtualization kvm. [ 3.312291] systemd[1]: Detected architecture x86-64. [ 3.314473] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.338684] systemd[1]: No hostname configured. [ 3.340587] systemd[1]: Set hostname to . [ 3.343832] random: systemd: uninitialized urandom read (16 bytes read) [ 3.346503] systemd[1]: Initializing machine ID from random generator. [ 3.510493] random: systemd: uninitialized urandom read (16 bytes read) [ 3.514981] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.522217] random: systemd: uninitialized urandom read (16 bytes read) [ 3.525134] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.531281] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.121646] device-mapper: uevent: version 1.0.3 [ 4.123836] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. [ 4.511354] random: fast init done Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.889368] virtio_net virtio0 ens2: renamed from eth0 [ 5.009693] scsi host0: ata_piix [ 5.018133] scsi host1: ata_piix [ 5.021069] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.024173] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.656842] dracut-initqueue[589]: RTNETLINK answers: File exists [ 9.747797] random: crng init done [ 9.749679] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.170737] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.293824] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.561177] SELinux: Disabled at runtime. [ 11.619764] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.632621] systemd[1]: Detected virtualization kvm. [ 11.635521] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.109064] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.112296] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.116385] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.122256] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.126730] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.133873] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.138265] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ 12.302130] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.578530] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.866336] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.923861] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.970145] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.991645] EDAC sbridge: Ver: 1.1.2 [ 14.045133] Key type dns_resolver registered [ 14.346097] NFS: Registering the id_resolver key type [ 14.347685] Key type id_resolver registered [ 14.349110] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg309-client login: [ 41.892730] hrtimer: interrupt took 7720888 ns [ 62.218167] libcfs: loading out-of-tree module taints kernel. [ 62.295946] Key type ._llcrypt registered [ 62.300248] Key type .llcrypt registered [ 63.027324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 63.042460] alg: No test for adler32 (adler32-zlib) [ 64.361232] Lustre: Lustre: Build Version: 2.17.58_2_g97ad22e [ 65.321467] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 67.095154] Key type lgssc registered [ 68.794208] Lustre: Echo OBD driver; http://www.lustre.org/ [ 186.337141] Lustre: Mounted lustre-client - version 2.17.58_2_g97ad22e [ 190.955444] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 203.629530] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing check_logdir /tmp/testlogs/ [ 208.159570] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing yml_node [ 211.935301] Lustre: lustre-OST0000-osc-ffff900c50d8c800: disconnect after 24s idle [ 212.260289] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 214.402609] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 216.672075] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 217.950793] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sat Sep 5 02:02:20 EDT 2026 [ 233.038529] Lustre: Unmounted lustre-client [ 261.522579] Key type lgssc unregistered [ 261.811568] LNet: 8439:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 261.821815] LNetError: 8439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 261.844161] LNet: Removed LNI 192.168.203.9@tcp [ 262.657486] Key type .llcrypt unregistered [ 262.663585] Key type ._llcrypt unregistered [ 285.219365] Key type ._llcrypt registered [ 285.222369] Key type .llcrypt registered [ 285.388107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 285.400298] alg: No test for adler32 (adler32-zlib) [ 286.269168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 286.333047] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 286.352214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 289.926955] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 290.020724] LNet: 9309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 290.028639] LNetError: 9309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 290.050633] LNet: Removed LNI 192.168.203.9@tcp [ 290.569539] Key type .llcrypt unregistered [ 290.572303] Key type ._llcrypt unregistered [ 296.156187] Key type ._llcrypt registered [ 296.159993] Key type .llcrypt registered [ 297.323190] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 02:03:39 (1788588219) [ 297.652272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 297.666992] alg: No test for adler32 (adler32-zlib) [ 298.590682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 298.651701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 298.664593] LNet: 9725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 298.685253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 298.734868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 298.762728] LNet: 9732:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 304.623309] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 02:03:46 (1788588226) [ 304.811529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 304.843253] LNet: 10307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 304.886654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 304.981834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 311.544103] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 02:03:53 (1788588233) [ 311.656464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 311.697323] LNet: 10888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 311.755652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 311.896439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 318.049673] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 02:04:00 (1788588240) [ 318.306706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 318.334265] LNet: 11478:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 318.357095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 318.424312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 323.950579] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 02:04:06 (1788588246) [ 324.028530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 324.043378] LNet: 12056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 324.062628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 324.163667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 324.419269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 330.884341] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 02:04:12 (1788588252) [ 331.073510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 331.112490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 331.185988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 331.582855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 332.013118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 338.141476] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 02:04:20 (1788588260) [ 338.210426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 338.237116] LNet: 13244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 338.242066] LNet: 13244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 338.258736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.365309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 344.487721] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 02:04:26 (1788588266) [ 344.562676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 344.589286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 344.656747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 344.684746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 345.030929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 345.080533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 345.419320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 345.455223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 345.874880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 345.913480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 346.281387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 346.348718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 346.757609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 346.796559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 347.306701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 347.332848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 347.722753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 347.748616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 348.125107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 348.167392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 348.666526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 348.722957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 349.214490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 349.251890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 349.720327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 349.792785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 350.300736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 350.341761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 356.860444] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 02:04:38 (1788588278) [ 357.029836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 357.081221] LNet: 14515:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 357.088649] LNet: 14515:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 357.112972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 357.186296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 357.233134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 363.700792] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 02:04:45 (1788588285) [ 363.809941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 363.851280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.905149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 363.933290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 370.218905] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 02:04:52 (1788588292) [ 370.503650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 370.549691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.639536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 370.661061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 377.234857] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 02:04:58 (1788588298) [ 377.586949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 377.652969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 377.760804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 377.827943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 385.474870] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 02:05:07 (1788588307) [ 385.638567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 385.718989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 385.812973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 385.868761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 393.300961] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 02:05:15 (1788588315) [ 393.388344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 393.409643] LNet: 17440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 393.415917] LNet: 17440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 393.436994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.559364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 393.599801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 400.116831] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 02:05:22 (1788588322) [ 400.390704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.445744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.546389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 401.008645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 406.929656] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 02:05:28 (1788588328) [ 407.222521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 407.249840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 407.509969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 407.723986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 414.153974] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 02:05:36 (1788588336) [ 414.584937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.630623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 414.769773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 414.944606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 421.780376] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 02:05:43 (1788588343) [ 421.888500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.920953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.357663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 422.571578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 429.407982] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 02:05:51 (1788588351) [ 429.699459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.742277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.810844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 429.866788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 436.065120] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 02:05:58 (1788588358) [ 436.135743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 436.178075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 436.250244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 436.308772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 442.328700] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 02:06:04 (1788588364) [ 442.500063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 442.543923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 442.618195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 449.056232] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 02:06:10 (1788588370) [ 449.152094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 449.223114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 449.315446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 456.032419] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 02:06:17 (1788588377) [ 456.311294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 456.359761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 456.424652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 457.007874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 464.044266] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 02:06:25 (1788588385) [ 464.219546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.258924] LNet: 23475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 464.268663] LNet: 23475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 464.302870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.428268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 464.500894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 471.338386] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 02:06:33 (1788588393) [ 471.597983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 471.668957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 471.811609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 471.872719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 478.360804] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 02:06:40 (1788588400) [ 478.647439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 478.696320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 478.790196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 478.829798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 486.071922] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 02:06:48 (1788588408) [ 486.562058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.602549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.694886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 486.741993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 493.448592] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 02:06:55 (1788588415) [ 493.788839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 493.832906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 493.903378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 493.963633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 493.988662] LNetError: 25833:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 494.059800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 494.117704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 500.788513] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 02:07:02 (1788588422) [ 501.031664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 501.112864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 501.227871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.ItB0qyJ9GH [ 501.329615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.ItB0qyJ9GH [ 501.414369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.ItB0qyJ9GH [ 508.422399] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 02:07:10 (1788588430) [ 508.519669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 508.569153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 508.636726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 508.705820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 508.748710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 515.852220] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 02:07:17 (1788588437) [ 516.216919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 516.277128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 516.347377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 516.365987] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 517.856284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 519.465290] Lustre: Lustre: Build Version: 2.17.58_2_g97ad22e [ 521.448040] Key type lgssc registered [ 522.816893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 526.564763] LNetError: 27595:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.203.9@tcp99 has no available nets [ 529.153318] Key type lgssc unregistered [ 529.564842] LNetError: 28700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 529.583683] LNet: Removed LNI 192.168.203.9@tcp [ 530.187125] Key type .llcrypt unregistered [ 530.189379] Key type ._llcrypt unregistered [ 537.122395] Key type ._llcrypt registered [ 537.125135] Key type .llcrypt registered [ 538.341254] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 02:07:40 (1788588460) [ 538.837618] Key type .llcrypt unregistered [ 538.842985] Key type ._llcrypt unregistered [ 546.190434] Key type ._llcrypt registered [ 546.196225] Key type .llcrypt registered [ 546.338548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 546.361423] alg: No test for adler32 (adler32-zlib) [ 547.524120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.644960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 547.709611] LNet: 29609:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 547.729032] LNet: 29609:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 547.750638] LNet: 29609:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 547.779242] LNet: Message forwarding will be enabled in 2s [ 550.879151] LNet: Message forwarding enabled [ 551.375838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 551.396947] LNet: Message forwarding will stop in 2s to 6s [ 554.977550] LNet: Message forwarding disabled [ 556.016249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 556.034757] LNet: 29708:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 556.049716] LNet: 29708:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 556.062936] LNet: 29708:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 556.082860] LNet: Message forwarding will be enabled in 2s [ 556.260098] LNet: 29721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 556.905522] Key type .llcrypt unregistered [ 556.908077] Key type ._llcrypt unregistered [ 569.383383] Key type ._llcrypt registered [ 569.387969] Key type .llcrypt registered [ 570.457743] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 02:08:12 (1788588492) [ 570.901125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 570.916639] alg: No test for adler32 (adler32-zlib) [ 571.821952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 571.892904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 577.412682] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 02:08:19 (1788588499) [ 577.639657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 577.656880] LNet: 31121:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 577.673199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 577.717426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 577.750787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 577.781229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 577.808946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 577.834966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 577.869406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 577.896836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 577.925347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 577.959113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 578.001766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 578.040498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 578.084485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 578.103280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 578.124537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 578.159065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 578.197458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 578.238209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 578.289384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 578.332502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 578.358558] LNetError: 31164:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 585.040321] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 02:08:26 (1788588506) [ 585.136773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 585.190784] LNet: 31727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 585.238067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 585.470313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 593.581470] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 02:08:35 (1788588515) [ 595.590699] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg309-server [ 597.412753] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 02:08:39 (1788588519) [ 597.778283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 597.799431] LNet: 32666:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 597.816934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 597.862428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 597.914358] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 599.021921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[10-14]@tcp [ 599.046921] LNet: 32799:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 599.103104] LNetError: 32676:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.12@tcp has gone from up to down [ 599.141612] LNetError: 32670:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.13@tcp is being used as a gateway but routing feature is not turned on [ 606.558657] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 02:08:48 (1788588528) [ 608.370755] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg309-server [ 609.752582] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 02:08:51 (1788588531) [ 610.151300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.194783] LNet: 33734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 610.208072] LNetError: 33734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 610.219860] LNet: Removed LNI 192.168.203.9@tcp [ 610.232199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.265788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 610.281115] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 610.893670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[10-15/2]@tcp [ 610.924172] LNet: 33852:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 610.938916] LNet: 33852:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 610.957543] LNetError: 33743:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.12@tcp has gone from up to down [ 610.968082] LNetError: 33743:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 610.974598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[10-15/2]@tcp [ 611.013806] LNetError: 33738:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.10@tcp is being used as a gateway but routing feature is not turned on [ 611.033433] LNetError: 33738:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 618.254270] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 02:08:59 (1788588539) [ 618.376208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 618.428482] LNet: 34439:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 618.447616] LNetError: 34439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 618.486050] LNet: Removed LNI 192.168.203.9@tcp [ 618.565168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 618.650875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 618.716776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 618.776626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 618.863200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 618.932483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 618.980436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.019927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 619.118684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 619.155399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.226714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 619.323284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 619.381597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.464447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 619.539910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 619.575368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.651830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 619.764977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 619.805894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.881650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 620.002813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.062155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.118934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 620.230592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.297689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.362967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 620.440548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.489233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.562934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 620.632651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.669874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.728449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 620.813208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.843833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.900671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 626.523438] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 02:09:08 (1788588548) [ 628.894136] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg309-server [ 630.530651] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 02:09:12 (1788588552) [ 632.363501] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg309-server [ 634.205420] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 02:09:16 (1788588556) [ 634.343108] LNet: 35836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 634.346456] LNet: 35836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 635.064507] Key type .llcrypt unregistered [ 635.070315] Key type ._llcrypt unregistered [ 642.506638] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 642.742960] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 642.745729] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 642.756782] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 645.574704] Key type ._llcrypt registered [ 645.576952] Key type .llcrypt registered [ 645.821906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 645.853336] alg: No test for adler32 (adler32-zlib) [ 646.930979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.029037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 647.119794] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 648.180068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 648.201845] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 650.349946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 650.371273] LNet: Removed LNI 10.1.2.3@tcp [ 650.587622] LNet: 36564:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 650.598410] LNetError: 36564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 650.613490] LNet: Removed LNI 192.168.203.9@tcp [ 651.349630] Key type .llcrypt unregistered [ 651.352343] Key type ._llcrypt unregistered [ 663.484701] Key type ._llcrypt registered [ 663.488398] Key type .llcrypt registered [ 664.474432] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 02:09:46 (1788588586) [ 665.437923] Key type .llcrypt unregistered [ 665.446420] Key type ._llcrypt unregistered [ 672.159544] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 672.241827] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 672.246318] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 672.251113] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 674.889877] Key type ._llcrypt registered [ 674.895084] Key type .llcrypt registered [ 675.062526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 675.081565] alg: No test for adler32 (adler32-zlib) [ 675.986189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.033083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 676.053895] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 676.777485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 676.794733] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 679.440824] LNet: 38055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 679.445964] LNetError: 38055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 679.450809] LNet: Removed LNI 10.1.2.3@tcp [ 679.458363] LNet: Removed LNI 192.168.203.9@tcp [ 680.081323] Key type .llcrypt unregistered [ 680.083717] Key type ._llcrypt unregistered [ 691.073755] Key type ._llcrypt registered [ 691.076413] Key type .llcrypt registered [ 692.339591] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 02:10:14 (1788588614) [ 692.941684] Key type .llcrypt unregistered [ 692.945902] Key type ._llcrypt unregistered [ 699.252189] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 699.465641] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 699.471935] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 699.480700] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 702.343835] Key type ._llcrypt registered [ 702.346336] Key type .llcrypt registered [ 702.624523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 702.646880] alg: No test for adler32 (adler32-zlib) [ 703.627482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 703.784596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 703.838731] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 707.014405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 707.061349] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 710.401911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 710.437076] LNet: Removed LNI 10.1.2.3@tcp [ 710.482367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 710.541692] LNet: Removed LNI 10.1.2.31@tcp [ 710.553679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 710.571056] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 711.442351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 711.473188] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 712.502212] LNet: 39969:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 712.515518] LNetError: 39969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 712.530149] LNet: Removed LNI 10.1.2.31@tcp [ 712.559430] LNet: Removed LNI 10.1.2.3@tcp [ 713.277936] Key type .llcrypt unregistered [ 713.281597] Key type ._llcrypt unregistered [ 728.094561] Key type ._llcrypt registered [ 728.097426] Key type .llcrypt registered [ 729.684780] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 02:10:51 (1788588651) [ 730.236329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 730.252410] alg: No test for adler32 (adler32-zlib) [ 731.098619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 731.155285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 731.179639] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 731.927512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 731.961927] LNet: 40920:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 731.969263] LNetError: 40920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 732.000150] LNet: Removed LNI 192.168.203.9@tcp [ 732.028218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 732.095936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.9@tcp [ 732.128308] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 732.866309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.9@tcp [ 732.906689] LNet: Removed LNI 192.168.203.9@tcp [ 732.953946] LNet: 41055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 733.626772] Key type .llcrypt unregistered [ 733.628970] Key type ._llcrypt unregistered [ 745.005660] Key type ._llcrypt registered [ 745.012619] Key type .llcrypt registered [ 746.488073] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 02:11:08 (1788588668) [ 748.386558] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg309-server [ 750.252487] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 02:11:11 (1788588671) [ 750.750331] Key type .llcrypt unregistered [ 750.755110] Key type ._llcrypt unregistered [ 756.879081] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 757.039635] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 757.044980] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 757.048742] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 759.662314] Key type ._llcrypt registered [ 759.665467] Key type .llcrypt registered [ 759.851057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 759.863580] alg: No test for adler32 (adler32-zlib) [ 760.810572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.902990] LNet: Added LNI 192.168.203.9@tcp1 [8/256/0/180] [ 761.596809] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 764.252996] LNet: 42887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 764.265636] LNetError: 42887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 764.275491] LNet: Removed LNI 192.168.203.9@tcp1 [ 764.299590] LNet: Removed LNI 10.1.2.3@tcp2 [ 764.999726] Key type .llcrypt unregistered [ 765.004305] Key type ._llcrypt unregistered [ 778.883487] Key type ._llcrypt registered [ 778.886261] Key type .llcrypt registered [ 780.460900] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 02:11:41 (1788588701) [ 780.938894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 780.956691] alg: No test for adler32 (adler32-zlib) [ 782.009583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 782.117604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 782.170407] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 783.734229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 783.775345] LNet: 43857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 783.796547] LNetError: 43857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 783.819048] LNet: Removed LNI 192.168.203.9@tcp [ 783.861847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 783.965513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 784.026423] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 791.647779] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 02:11:53 (1788588713) [ 791.746510] LNet: 44443:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 791.765573] LNetError: 44443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 791.781049] LNet: Removed LNI 192.168.203.9@tcp [ 792.327556] Key type .llcrypt unregistered [ 792.335077] Key type ._llcrypt unregistered [ 798.981397] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 799.201679] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 799.209946] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 799.213933] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 802.142446] Key type ._llcrypt registered [ 802.144377] Key type .llcrypt registered [ 802.333719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 802.347480] alg: No test for adler32 (adler32-zlib) [ 803.219154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 803.275575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 803.309662] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 804.172977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 804.202601] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 807.252908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 807.280324] LNet: 45187:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 807.286667] LNetError: 45187:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 807.297321] LNet: Removed LNI 10.1.2.3@tcp [ 807.327594] LNet: Removed LNI 192.168.203.9@tcp [ 807.376321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 807.484519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 807.525644] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 807.539195] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 808.513647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 808.562218] LNet: 45214:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 808.575932] LNetError: 45214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 808.604087] LNetError: 45214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 808.634102] LNet: Removed LNI 10.1.2.3@tcp [ 808.651147] LNet: Removed LNI 192.168.203.9@tcp [ 808.698595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 808.842277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 808.877828] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 808.882831] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 809.772864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 809.829210] LNet: Added LNI 192.168.203.9@tcp2 [8/256/0/180] [ 811.320491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 811.384102] LNet: 45377:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 811.394604] LNetError: 45377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 811.402617] LNetError: 45377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 811.415447] LNet: Removed LNI 10.1.2.3@tcp [ 811.424341] LNet: Removed LNI 192.168.203.9@tcp [ 811.447490] LNet: Removed LNI 192.168.203.9@tcp2 [ 811.496482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.637482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 811.704925] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 811.717361] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 811.723896] LNet: Added LNI 192.168.203.9@tcp2 [8/256/0/180] [ 820.808370] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 02:12:22 (1788588742) [ 820.946698] LNet: 45970:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.953914] LNetError: 45970:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 820.958500] LNetError: 45970:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 820.963496] LNet: Removed LNI 10.1.2.3@tcp [ 820.965510] LNet: Removed LNI 192.168.203.9@tcp [ 820.977792] LNet: Removed LNI 192.168.203.9@tcp2 [ 821.623514] Key type .llcrypt unregistered [ 821.625523] Key type ._llcrypt unregistered [ 829.586735] Key type ._llcrypt registered [ 829.589352] Key type .llcrypt registered [ 829.837991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 829.858864] alg: No test for adler32 (adler32-zlib) [ 830.930448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 831.042878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 831.106878] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 832.192964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 832.215142] LNet: 46366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 832.226664] LNetError: 46366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 832.250969] LNet: Removed LNI 192.168.203.9@tcp [ 832.296181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 832.360108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 832.387071] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 833.009151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 833.026220] LNet: 46396:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 833.032766] LNetError: 46396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 833.047697] LNet: Removed LNI 192.168.203.9@tcp [ 833.104651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 833.226324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 833.258816] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 833.945920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 833.981158] LNet: Removed LNI 192.168.203.9@tcp [ 834.004603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 834.074330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 834.112435] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 834.660362] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 834.826895] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 834.832585] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 834.838338] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 837.209306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 837.232574] LNet: 46502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 837.238028] LNet: 46502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 837.246427] LNetError: 46502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 837.252884] LNetError: 46502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 837.266755] LNet: Removed LNI 192.168.203.9@tcp [ 837.278524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 837.326739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 837.344214] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 844.040098] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 02:12:45 (1788588765) [ 844.157037] LNet: 47090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.175074] LNetError: 47090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 844.199074] LNet: Removed LNI 192.168.203.9@tcp [ 844.944838] Key type .llcrypt unregistered [ 844.950470] Key type ._llcrypt unregistered [ 850.519898] Key type ._llcrypt registered [ 850.523507] Key type .llcrypt registered [ 850.685086] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 850.704541] alg: No test for adler32 (adler32-zlib) [ 851.593795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 851.667535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 851.701660] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 852.508849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 852.536193] LNet: 47494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 852.540988] LNetError: 47494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 852.552672] LNet: Removed LNI 192.168.203.9@tcp [ 852.572377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 852.633311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 852.659392] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 858.212393] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 02:13:00 (1788588780) [ 858.339097] LNet: 48082:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.347978] LNetError: 48082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 858.373944] LNet: Removed LNI 192.168.203.9@tcp [ 858.989748] Key type .llcrypt unregistered [ 858.993161] Key type ._llcrypt unregistered [ 863.884491] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 863.989118] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 863.995768] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 864.000301] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 866.587721] Key type ._llcrypt registered [ 866.590832] Key type .llcrypt registered [ 866.777867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 866.796277] alg: No test for adler32 (adler32-zlib) [ 867.805112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 867.868862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 867.897524] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 868.624494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 868.654745] LNet: 48531:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 868.663899] LNetError: 48531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 868.676784] LNet: Removed LNI 192.168.203.9@tcp [ 868.700735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 868.790513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 868.824797] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 876.074469] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 02:13:18 (1788588798) [ 876.532090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 876.571140] LNet: 49130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 876.586157] LNetError: 49130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 876.601932] LNet: Removed LNI 192.168.203.9@tcp [ 876.617232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 876.809388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 883.492077] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 02:13:25 (1788588805) [ 883.928027] LNet: 49713:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 884.642849] Key type .llcrypt unregistered [ 884.644874] Key type ._llcrypt unregistered [ 890.709674] Key type ._llcrypt registered [ 890.712373] Key type .llcrypt registered [ 890.908043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 890.925225] alg: No test for adler32 (adler32-zlib) [ 891.879384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 891.975780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 892.017666] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 892.905551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 892.926607] LNet: 50107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 892.934923] LNetError: 50107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 892.950360] LNet: Removed LNI 192.168.203.9@tcp [ 892.983750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 893.079815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 893.106470] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 899.791828] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 02:13:41 (1788588821) [ 900.081920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 900.102605] LNet: 50700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.114763] LNetError: 50700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 900.159768] LNet: Removed LNI 192.168.203.9@tcp [ 900.226231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 900.315672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 906.665158] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 02:13:48 (1788588828) [ 908.198940] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 910.015956] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 02:13:51 (1788588831) [ 911.923045] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 913.667754] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 02:13:55 (1788588835) [ 915.039653] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 916.668327] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 02:13:58 (1788588838) [ 918.363483] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 920.333819] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 02:14:02 (1788588842) [ 922.069768] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 923.909615] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 02:14:05 (1788588845) [ 924.374688] LNet: 52754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 925.233903] Key type .llcrypt unregistered [ 925.240086] Key type ._llcrypt unregistered [ 931.656145] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 931.861603] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 931.868652] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 931.873264] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 934.477893] Key type ._llcrypt registered [ 934.481092] Key type .llcrypt registered [ 934.697769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 934.718329] alg: No test for adler32 (adler32-zlib) [ 935.689779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 935.779280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 935.799158] LNet: Added LNI 192.168.203.9@tcp [8/512/0/180] [ 935.804811] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 935.807729] LNet: 53066:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 935.878859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 935.894386] LNet: 53085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 935.898519] LNetError: 53085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 935.903659] LNet: Removed LNI 10.1.2.3@tcp [ 935.907613] LNetError: 53085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 935.917863] LNet: Removed LNI 192.168.203.9@tcp [ 935.940675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 936.012682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 936.036188] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 936.094458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 936.108673] LNet: 53109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 936.122440] LNetError: 53109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 936.139764] LNet: Removed LNI 192.168.203.9@tcp [ 936.159031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 936.240490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 936.286945] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 936.293567] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 936.299810] LNet: 53117:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 936.446504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 936.476754] LNet: 53135:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 936.483928] LNetError: 53135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 936.490611] LNet: Removed LNI 10.1.2.3@tcp [ 936.493956] LNetError: 53135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 936.510502] LNet: Removed LNI 192.168.203.9@tcp [ 936.533851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 936.599463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 936.630032] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 936.638384] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 936.641844] LNet: 53143:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 936.847814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 936.864857] LNet: 53169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 936.871604] LNetError: 53169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 936.876774] LNet: Removed LNI 10.1.2.3@tcp [ 936.878985] LNetError: 53169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 936.892357] LNet: Removed LNI 192.168.203.9@tcp [ 936.928470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 937.004467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 937.053730] LNet: Added LNI 192.168.203.9@tcp [8/256/0/182] [ 937.066311] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 937.218809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 937.239809] LNet: 53197:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 937.245237] LNetError: 53197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 937.250564] LNet: Removed LNI 10.1.2.3@tcp [ 937.253976] LNetError: 53197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 937.266281] LNet: Removed LNI 192.168.203.9@tcp [ 937.306567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 937.416791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 937.456393] LNet: Added LNI 192.168.203.9@tcp [8/256/0/182] [ 937.469978] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 937.474721] LNet: 53205:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 937.630703] LNet: Removed LNI 10.1.2.3@tcp [ 937.673247] LNet: Removed LNI 192.168.203.9@tcp [ 938.485120] Key type .llcrypt unregistered [ 938.488536] Key type ._llcrypt unregistered [ 949.488163] Key type ._llcrypt registered [ 949.489789] Key type .llcrypt registered [ 950.515418] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 02:14:32 (1788588872) [ 951.730153] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 953.708036] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 02:14:35 (1788588875) [ 954.354106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 954.378530] alg: No test for adler32 (adler32-zlib) [ 955.302061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 955.338401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 955.372585] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 956.580737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.9 [ 956.618342] LNet: 54468:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 956.632098] LNet: Removed LNI 192.168.203.9@tcp [ 966.817822] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 02:14:48 (1788588888) [ 967.129279] LNet: 55032:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 967.159040] LNetError: 55032:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 967.211551] LNet: Removed LNI 192.168.203.9@tcp [ 968.515395] Key type .llcrypt unregistered [ 968.518660] Key type ._llcrypt unregistered [ 978.589736] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 978.872524] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 981.658017] Key type ._llcrypt registered [ 981.659872] Key type .llcrypt registered [ 981.771038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 981.779797] alg: No test for adler32 (adler32-zlib) [ 983.044289] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 990.719660] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 02:15:12 (1788588912) [ 991.206852] LNet: 55952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 991.221784] LNetError: 55952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 991.241389] LNet: Removed LNI 10.1.2.3@tcp [ 991.902470] Key type .llcrypt unregistered [ 991.907233] Key type ._llcrypt unregistered [ 997.493289] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 997.798422] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 997.811548] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 997.819108] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1000.772476] Key type ._llcrypt registered [ 1000.774178] Key type .llcrypt registered [ 1000.979472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1001.015497] alg: No test for adler32 (adler32-zlib) [ 1002.073701] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1008.218828] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 02:15:30 (1788588930) [ 1009.570794] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1011.194527] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 02:15:33 (1788588933) [ 1011.398753] LNet: 57129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1011.406898] LNetError: 57129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1011.446781] LNet: Removed LNI 10.1.2.3@tcp [ 1012.233590] Key type .llcrypt unregistered [ 1012.235251] Key type ._llcrypt unregistered [ 1019.477503] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1019.777839] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1019.785365] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1019.794426] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1022.656848] Key type ._llcrypt registered [ 1022.659271] Key type .llcrypt registered [ 1022.905447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1022.925760] alg: No test for adler32 (adler32-zlib) [ 1023.864260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1023.944096] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1031.985202] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 02:15:53 (1788588953) [ 1032.473859] LNet: 58359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1032.480756] LNetError: 58359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1032.512975] LNet: Removed LNI 10.1.2.3@tcp [ 1033.188560] Key type .llcrypt unregistered [ 1033.191293] Key type ._llcrypt unregistered [ 1040.759810] Key type ._llcrypt registered [ 1040.763764] Key type .llcrypt registered [ 1041.008908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1041.026840] alg: No test for adler32 (adler32-zlib) [ 1042.063489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1042.111226] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1051.307816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.109@tcp [ 1052.085748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e local_interrupt -r 1 [ 1052.163988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e local_interrupt -r 1 [ 1052.227527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.109@tcp [ 1052.265269] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1052.384189] LNet: 1 local NIs in recovery (showing 1): 192.168.203.9@tcp [ 1052.466424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1052.487619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1052.510847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1052.543603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1052.563996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1052.583180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1052.603388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1052.638143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1052.684768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1052.710555] LNet: 58796:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1052.734843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1052.758962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1052.784283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1052.805473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1052.823733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1052.844768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1052.866216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1053.339219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e local_dropped -r 1 [ 1053.392611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e local_dropped -r 1 [ 1053.446084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.109@tcp [ 1053.477898] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1053.775075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1053.829848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1053.879644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1053.937486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1053.979839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1054.023117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1054.065170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1054.110645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1054.141135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1054.167151] LNet: 58907:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1054.194921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1054.241896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1054.289522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1054.333297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1054.392125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1054.443437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1054.491081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1055.168695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e local_aborted -r 1 [ 1055.229945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e local_aborted -r 1 [ 1055.314068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.109@tcp [ 1055.349973] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1055.362758] LNet: Skipped 1 previous similar message [ 1055.636514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1055.669941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1055.704254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1055.744517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1055.788685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1055.834687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1055.874175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1055.910751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1055.933776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1055.973247] LNet: 59018:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1055.999814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1056.049586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1056.089534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1056.152994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1056.182574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1056.215279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1056.245808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1056.827100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e local_no_route -r 1 [ 1056.854621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e local_no_route -r 1 [ 1056.904694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.109@tcp [ 1057.143904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1057.179858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1057.205860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1057.224301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1057.261561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1057.288281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1057.321643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1057.352029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1057.383775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1057.403906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1057.432234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1057.459962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1057.486559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1057.509677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1057.537463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1057.585763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1058.183262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e local_timeout -r 1 [ 1058.238530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e local_timeout -r 1 [ 1058.262363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.109@tcp [ 1058.273938] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1058.281098] LNet: Skipped 1 previous similar message [ 1058.464563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1058.497812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1058.528859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1058.554854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1058.583106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1058.607203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1058.635295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1058.672524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1058.726668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1058.755117] LNet: 59241:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1058.763057] LNet: 59241:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1058.783996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1058.812226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1058.851146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1058.901471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1058.936510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1058.962189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1058.998481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1059.580470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e local_error -r 1 [ 1059.615734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e local_error -r 1 [ 1059.657206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.109@tcp [ 1059.922755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1059.967938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1060.019180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1060.058709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1060.084595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1060.114382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1060.142511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1060.203925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1060.242271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1060.278887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1060.318990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1060.363584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1060.394910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1060.420471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1060.459993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1060.491947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1060.757800] LNet: 59393:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1060.770713] LNetError: 59393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1060.788427] LNet: Removed LNI 192.168.203.9@tcp [ 1061.288652] Key type .llcrypt unregistered [ 1061.290419] Key type ._llcrypt unregistered [ 1078.395389] Key type ._llcrypt registered [ 1078.398473] Key type .llcrypt registered [ 1079.592318] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 02:16:41 (1788589001) [ 1080.461588] Key type .llcrypt unregistered [ 1080.464074] Key type ._llcrypt unregistered [ 1086.037716] Key type ._llcrypt registered [ 1086.046071] Key type .llcrypt registered [ 1086.224508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1086.237935] alg: No test for adler32 (adler32-zlib) [ 1087.146191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1087.170908] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1094.969840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.109@tcp [ 1106.834869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1106.892625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1106.921644] LNet: Added LNI 192.168.203.9@tcp1 [8/256/0/180] [ 1108.996980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e local_interrupt -r 1 [ 1109.059735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp1 -m GET -e local_interrupt -r 1 [ 1109.166719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e local_interrupt -r 1 [ 1109.220463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp1 -m GET -e local_interrupt -r 1 [ 1109.262682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp -m GET -e local_interrupt -r 1 [ 1109.297257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp1 -m GET -e local_interrupt -r 1 [ 1109.360940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp -m GET -e local_interrupt -r 1 [ 1109.431442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp1 -m GET -e local_interrupt -r 1 [ 1109.493415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.109@tcp [ 1109.510920] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1109.518340] LNet: 1 local NIs in recovery (showing 1): 192.168.203.9@tcp [ 1109.872821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1109.923613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1109.960708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1109.993699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1110.025730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1110.071216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1110.105140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1110.145784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1110.176475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1110.212756] LNet: 60861:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1110.251699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1110.333435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1110.395918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1110.454625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1110.492533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1110.558143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1110.577865] LNet: There was an unexpected network error while writing to 192.168.203.9: rc = -22 [ 1110.599464] LNet: Skipped 3 previous similar messages [ 1110.638966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1111.271889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e local_dropped -r 1 [ 1111.374214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp1 -m GET -e local_dropped -r 1 [ 1111.434515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e local_dropped -r 1 [ 1111.474685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp1 -m GET -e local_dropped -r 1 [ 1111.532987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp -m GET -e local_dropped -r 1 [ 1111.571335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp1 -m GET -e local_dropped -r 1 [ 1111.641191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp -m GET -e local_dropped -r 1 [ 1111.695811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp1 -m GET -e local_dropped -r 1 [ 1111.772848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.109@tcp [ 1111.798374] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1112.118400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1112.156188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1112.198217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1112.246394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1112.301317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1112.356257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1112.414155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1112.448916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1112.482648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1112.503243] LNet: 60984:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1112.535199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1112.570184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1112.608411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1112.643857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1112.712198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1112.763507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1112.800677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1113.683552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e local_aborted -r 1 [ 1113.763220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp1 -m GET -e local_aborted -r 1 [ 1113.816644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e local_aborted -r 1 [ 1113.866803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp1 -m GET -e local_aborted -r 1 [ 1113.900654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp -m GET -e local_aborted -r 1 [ 1113.936499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp1 -m GET -e local_aborted -r 1 [ 1113.980452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp -m GET -e local_aborted -r 1 [ 1114.039997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp1 -m GET -e local_aborted -r 1 [ 1114.094581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.109@tcp [ 1114.123289] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1114.135575] LNet: Skipped 2 previous similar messages [ 1114.434439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1114.479180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1114.516986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1114.572780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1114.617303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1114.653744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1114.696923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1114.776428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1114.812748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1114.865733] LNet: 61108:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1114.904171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1114.987783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1115.046429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1115.084652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1115.125624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1115.164992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1115.247870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1115.991283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e local_no_route -r 1 [ 1116.060584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp1 -m GET -e local_no_route -r 1 [ 1116.120286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e local_no_route -r 1 [ 1116.185322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp1 -m GET -e local_no_route -r 1 [ 1116.228463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp -m GET -e local_no_route -r 1 [ 1116.271450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp1 -m GET -e local_no_route -r 1 [ 1116.313776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp -m GET -e local_no_route -r 1 [ 1116.357758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp1 -m GET -e local_no_route -r 1 [ 1116.403711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.109@tcp [ 1116.712667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1116.745862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1116.784717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1116.809419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1116.844880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1116.875274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1116.921330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1116.961550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1117.013530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1117.043353] LNet: 61231:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1117.066520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1117.096294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1117.201788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1117.256158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1117.289334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1117.317355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1117.341419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1117.859406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e local_timeout -r 1 [ 1117.906209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp1 -m GET -e local_timeout -r 1 [ 1117.959346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e local_timeout -r 1 [ 1118.037404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp1 -m GET -e local_timeout -r 1 [ 1118.101416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp -m GET -e local_timeout -r 1 [ 1118.147112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp1 -m GET -e local_timeout -r 1 [ 1118.213528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp -m GET -e local_timeout -r 1 [ 1118.267727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp1 -m GET -e local_timeout -r 1 [ 1118.324217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.109@tcp [ 1118.346975] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1118.357537] LNet: Skipped 5 previous similar messages [ 1118.596684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1118.637228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1118.668582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1118.711876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1118.748454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1118.778594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1118.818609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1118.856964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1118.893877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1118.933516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1118.958718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1118.987740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1119.032408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1119.061303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1119.094558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1119.120441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1119.744649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e local_error -r 1 [ 1119.783686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp1 -m GET -e local_error -r 1 [ 1119.819503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e local_error -r 1 [ 1119.863839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp1 -m GET -e local_error -r 1 [ 1119.903312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp -m GET -e local_error -r 1 [ 1119.950716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp1 -m GET -e local_error -r 1 [ 1119.983762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp -m GET -e local_error -r 1 [ 1120.013868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp1 -m GET -e local_error -r 1 [ 1120.096929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.109@tcp [ 1120.354901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1120.378911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1120.411342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1120.438694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1120.469898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1120.503893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1120.539191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1120.580172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1120.611472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1120.655779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1120.705066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1120.752621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1120.777656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1120.810350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1120.854228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1120.897537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1126.521348] LNet: 61554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1126.530978] LNetError: 61554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1127.584960] LNet: Removed LNI 192.168.203.9@tcp [ 1127.591557] LNetError: 61554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1127.610744] LNet: Removed LNI 192.168.203.9@tcp1 [ 1128.037615] Key type .llcrypt unregistered [ 1128.042845] Key type ._llcrypt unregistered [ 1146.063138] Key type ._llcrypt registered [ 1146.065451] Key type .llcrypt registered [ 1147.528917] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 02:17:49 (1788589069) [ 1148.352680] Key type .llcrypt unregistered [ 1148.358880] Key type ._llcrypt unregistered [ 1153.460700] Key type ._llcrypt registered [ 1153.464451] Key type .llcrypt registered [ 1153.646136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1153.658708] alg: No test for adler32 (adler32-zlib) [ 1154.563400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1154.595614] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1161.906582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.109@tcp [ 1162.667314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e remote_dropped -r 1 [ 1162.721913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e remote_dropped -r 1 [ 1162.770243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.109@tcp [ 1162.800437] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1163.040288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1163.079519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1163.106563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1163.141727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1163.174482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1163.202943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1163.242274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1163.274766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1163.357515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1163.387959] LNet: 62801:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1163.404508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1163.436630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1163.483867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1163.517431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1163.556552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1163.595242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1163.630626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1164.194907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e remote_error -r 1 [ 1164.247417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e remote_error -r 1 [ 1164.289465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.109@tcp [ 1164.310054] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1164.524301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1164.559443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1164.605240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1164.640805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1164.676754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1164.726380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1164.771241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1164.801767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1164.845423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1164.869240] LNet: 62913:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1164.896966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1164.935073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1164.975940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1165.012495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1165.040240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1165.069986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1165.127256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1165.679766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e remote_timeout -r 1 [ 1165.724555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e remote_timeout -r 1 [ 1165.754185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.109@tcp [ 1165.768936] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1165.980398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1166.008380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1166.032841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1166.059665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1166.123737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1166.168838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1166.208627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1166.253452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1166.284428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1166.313638] LNet: 63024:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1166.343292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1166.385408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1166.415941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1166.471646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1166.512306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1166.556853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1166.586920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1166.855166] LNet: 63065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1166.873702] LNetError: 63065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1167.920442] LNet: Removed LNI 192.168.203.9@tcp [ 1168.355442] Key type .llcrypt unregistered [ 1168.357497] Key type ._llcrypt unregistered [ 1185.602973] Key type ._llcrypt registered [ 1185.606197] Key type .llcrypt registered [ 1186.867289] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 02:18:28 (1788589108) [ 1187.769763] Key type .llcrypt unregistered [ 1187.772375] Key type ._llcrypt unregistered [ 1194.203482] Key type ._llcrypt registered [ 1194.206474] Key type .llcrypt registered [ 1194.474493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1194.496775] alg: No test for adler32 (adler32-zlib) [ 1195.425596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1195.453561] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1202.557398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.109@tcp [ 1211.079833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1211.105596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1211.130036] LNet: Added LNI 192.168.203.9@tcp1 [8/256/0/180] [ 1212.374483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e remote_dropped -r 1 [ 1212.414546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp1 -m GET -e remote_dropped -r 1 [ 1212.450474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e remote_dropped -r 1 [ 1212.484853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp1 -m GET -e remote_dropped -r 1 [ 1212.524952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp -m GET -e remote_dropped -r 1 [ 1212.573242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp1 -m GET -e remote_dropped -r 1 [ 1212.620323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp -m GET -e remote_dropped -r 1 [ 1212.674406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp1 -m GET -e remote_dropped -r 1 [ 1212.709936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.109@tcp [ 1212.725549] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1212.730638] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.109@tcp [ 1212.918564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1212.941965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1212.966851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1212.990636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1213.021538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1213.044776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1213.072169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1213.093682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1213.116573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1213.130722] LNet: 64531:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1213.145649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1213.171481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1213.197455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1213.225401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1213.248707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1213.287745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1213.321478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1213.360445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1213.760578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e remote_error -r 1 [ 1213.790860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp1 -m GET -e remote_error -r 1 [ 1213.832458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e remote_error -r 1 [ 1213.869887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp1 -m GET -e remote_error -r 1 [ 1213.902800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp -m GET -e remote_error -r 1 [ 1213.939958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp1 -m GET -e remote_error -r 1 [ 1213.972670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp -m GET -e remote_error -r 1 [ 1213.999616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp1 -m GET -e remote_error -r 1 [ 1214.037825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.109@tcp [ 1214.059926] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1214.070827] LNet: Skipped 2 previous similar messages [ 1214.270929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1214.294897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1214.314973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1214.335968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1214.364949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1214.405293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1214.439698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1214.473511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1214.505119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1214.538283] LNet: 64656:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1214.559399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1214.596073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1214.623154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1214.668045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1214.727385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1214.778155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1214.834090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1214.887946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1215.344245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e remote_timeout -r 1 [ 1215.386636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp1 -m GET -e remote_timeout -r 1 [ 1215.426720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e remote_timeout -r 1 [ 1215.463645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp1 -m GET -e remote_timeout -r 1 [ 1215.498277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp -m GET -e remote_timeout -r 1 [ 1215.532774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp1 -m GET -e remote_timeout -r 1 [ 1215.579571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp -m GET -e remote_timeout -r 1 [ 1215.644100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp1 -m GET -e remote_timeout -r 1 [ 1215.698354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.109@tcp [ 1215.721679] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1215.940843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1215.970823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1216.001189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1216.030388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1216.055988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1216.096086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1216.123250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1216.162986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1216.199896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1216.224729] LNet: 64782:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1216.242796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1216.280350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1216.327626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1216.356785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1216.402279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1216.432961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1216.463357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1216.493091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1221.018359] LNet: 64860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1221.041496] LNetError: 64860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1221.058924] LNet: Removed LNI 192.168.203.9@tcp [ 1221.078378] LNet: Removed LNI 192.168.203.9@tcp1 [ 1221.497498] Key type .llcrypt unregistered [ 1221.509053] Key type ._llcrypt unregistered [ 1237.384367] Key type ._llcrypt registered [ 1237.391970] Key type .llcrypt registered [ 1238.424761] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 02:19:20 (1788589160) [ 1239.743070] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1241.394403] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 02:19:23 (1788589163) [ 1242.260447] Key type .llcrypt unregistered [ 1242.265294] Key type ._llcrypt unregistered [ 1248.580864] Key type ._llcrypt registered [ 1248.583695] Key type .llcrypt registered [ 1248.809777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1248.838441] alg: No test for adler32 (adler32-zlib) [ 1249.858298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1249.930328] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1257.046472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.109@tcp [ 1257.965893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e network_timeout -r 1 [ 1258.006422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e network_timeout -r 1 [ 1258.046294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.109@tcp [ 1258.065919] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1258.144623] LNet: 1 local NIs in recovery (showing 1): 192.168.203.9@tcp [ 1258.325660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1258.374656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1258.407369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1258.445333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1258.479641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1258.507250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1258.532404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1258.557930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1258.587828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1258.602930] LNet: 66400:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1258.615731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1258.643480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1258.700401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1258.742186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1258.782369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1258.842319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1259.064836] LNet: 66439:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1259.075663] LNetError: 66439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1259.093847] LNet: Removed LNI 192.168.203.9@tcp [ 1259.452253] Key type .llcrypt unregistered [ 1259.454066] Key type ._llcrypt unregistered [ 1279.351830] Key type ._llcrypt registered [ 1279.354228] Key type .llcrypt registered [ 1279.562953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1279.575573] alg: No test for adler32 (adler32-zlib) [ 1280.492122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1280.542266] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1288.220785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.109@tcp [ 1297.265338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1297.322971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1297.353631] LNet: Added LNI 192.168.203.9@tcp1 [8/256/0/180] [ 1298.795534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e network_timeout -r 1 [ 1298.825668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp1 -m GET -e network_timeout -r 1 [ 1298.850854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e network_timeout -r 1 [ 1298.879490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp1 -m GET -e network_timeout -r 1 [ 1298.934640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp -m GET -e network_timeout -r 1 [ 1298.980657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp1 -m GET -e network_timeout -r 1 [ 1299.034452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp -m GET -e network_timeout -r 1 [ 1299.075438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp1 -m GET -e network_timeout -r 1 [ 1299.120181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.109@tcp [ 1299.162485] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1299.431088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1299.467289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1299.495178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1299.519540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1299.550370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1299.585887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1299.622560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1299.653506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1299.677332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1299.689877] LNet: 67317:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1299.705702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1299.760860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1299.813592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1299.844900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1299.891629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1299.948823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1305.016465] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1305.119102] LNet: 1 local NIs in recovery (showing 1): 192.168.203.9@tcp [ 1305.303298] LNet: 67391:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1305.311845] LNetError: 67391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1305.318052] LNet: Removed LNI 192.168.203.9@tcp [ 1305.339893] LNet: Removed LNI 192.168.203.9@tcp1 [ 1305.703526] Key type .llcrypt unregistered [ 1305.710447] Key type ._llcrypt unregistered [ 1323.649197] Key type ._llcrypt registered [ 1323.651313] Key type .llcrypt registered [ 1325.378066] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 02:20:46 (1788589246) [ 1326.039443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1326.061961] alg: No test for adler32 (adler32-zlib) [ 1327.212654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1327.299418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1327.334379] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1328.442745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1328.490874] LNet: Added LNI 192.168.203.9@tcp1 [8/256/0/180] [ 1329.643900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 1329.721632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1329.761796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1329.802543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1329.881917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1329.949937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1330.021939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.9@tcp [ 1330.066677] LNetError: 68490:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.9@tcp to recovery [ 1330.335414] LNet: 1 local NIs in recovery (showing 1): 192.168.203.9@tcp [ 1330.335684] LNet: There was an unexpected network error while writing to 192.168.203.9: rc = -22 [ 1332.384190] LNet: There was an unexpected network error while writing to 192.168.203.9: rc = -22 [ 1333.546621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1336.481782] LNet: There was an unexpected network error while writing to 192.168.203.9: rc = -22 [ 1337.309857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1337.382356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1337.544301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1337.640257] LNet: 68597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1337.669779] LNetError: 68597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1337.687071] LNet: Removed LNI 192.168.203.9@tcp [ 1337.710773] LNet: Removed LNI 192.168.203.9@tcp1 [ 1337.753717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1337.810794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1337.849306] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1339.096846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1339.114314] LNet: Added LNI 192.168.203.9@tcp1 [8/256/0/180] [ 1340.274427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 1340.340957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1340.392865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1340.458676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1340.492198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1340.533538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1340.558211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1340.591684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.9@tcp [ 1340.605397] LNetError: 68870:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.9@tcp to recovery [ 1340.895819] LNet: There was an unexpected network error while writing to 192.168.203.9: rc = -22 [ 1344.260844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1346.026082] LNet: There was an unexpected network error while writing to 192.168.203.9: rc = -22 [ 1346.043844] LNet: Skipped 1 previous similar message [ 1350.527275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1350.622293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1350.700289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1358.891918] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 02:21:20 (1788589280) [ 1359.040693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1359.062190] LNet: 69565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1359.077872] LNetError: 69565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1359.096430] LNetError: 69565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1359.110514] LNet: Removed LNI 192.168.203.9@tcp [ 1359.165737] LNet: Removed LNI 192.168.203.9@tcp1 [ 1359.226700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1359.308939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1359.332487] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1360.080496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1360.134270] LNet: Added LNI 192.168.203.9@tcp1 [8/256/0/180] [ 1361.105892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 1361.198567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1361.260977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1361.365435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1361.443513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1361.476429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1361.511195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.9@tcp --health 0 [ 1361.639572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1362.335288] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.9@tcp [ 1362.335726] LNet: There was an unexpected network error while writing to 192.168.203.9: rc = -22 [ 1362.352130] LNet: Skipped 1 previous similar message [ 1377.240790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1377.390557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1377.432208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1377.478601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.9@tcp --health 500 [ 1377.608775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1384.350399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1384.421703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1384.451492] LNet: 70061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1384.463513] LNetError: 70061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1384.470590] LNetError: 70061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1384.477840] LNet: Removed LNI 192.168.203.9@tcp [ 1384.489764] LNet: Removed LNI 192.168.203.9@tcp1 [ 1384.537942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1384.595860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1384.640196] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1385.460296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1385.492869] LNet: Added LNI 192.168.203.9@tcp1 [8/256/0/180] [ 1386.179955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 1386.235251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1386.322402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1386.391052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1386.469917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1386.558933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1386.609803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1386.651382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.9@tcp --health 0 [ 1387.680062] LNet: There was an unexpected network error while writing to 192.168.203.9: rc = -22 [ 1387.694030] LNet: Skipped 3 previous similar messages [ 1388.065924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1398.433802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1398.471994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1398.531118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1405.447381] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 02:22:07 (1788589327) [ 1405.798643] LNet: 71107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1405.803929] LNetError: 71107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1405.808901] LNetError: 71107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1405.817806] LNet: Removed LNI 192.168.203.9@tcp [ 1405.828579] LNet: Removed LNI 192.168.203.9@tcp1 [ 1406.413860] Key type .llcrypt unregistered [ 1406.418240] Key type ._llcrypt unregistered [ 1411.868512] Key type ._llcrypt registered [ 1411.871470] Key type .llcrypt registered [ 1412.041629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1412.053773] alg: No test for adler32 (adler32-zlib) [ 1412.961213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1412.979758] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1421.484865] Lustre: DEBUG MARKER: Initial discovery [ 1421.760219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.109@tcp [ 1423.901394] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1423.949238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1424.006995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.109@tcp [ 1424.029661] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1424.067750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1424.128403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.109@tcp -r 1 -m GET -l 3 [ 1424.183094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.109@tcp -r 1 -m GET -e local_error [ 1424.209829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.109@tcp -r 1 -m PUT -l 6 [ 1424.292237] LNet: 1 local NIs in recovery (showing 1): 192.168.203.9@tcp [ 1425.604786] Lustre: DEBUG MARKER: Force oleg309-client.virtnet to discover 192.168.203.109@tcp (in background) [ 1425.654566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.109@tcp [ 1426.982576] Lustre: DEBUG MARKER: Force oleg309-server to discover 192.168.203.9@tcp [ 1428.708738] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1429.309983] Lustre: DEBUG MARKER: Wait for 71859 [ 1430.985970] Lustre: DEBUG MARKER: Finished wait on 71859 [ 1431.032942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1431.064237] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1431.068658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1431.410862] LNet: 72317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1431.423467] LNetError: 72317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1431.441974] LNet: Removed LNI 192.168.203.9@tcp [ 1431.825631] Key type .llcrypt unregistered [ 1431.829524] Key type ._llcrypt unregistered [ 1445.192679] Key type ._llcrypt registered [ 1445.195464] Key type .llcrypt registered [ 1446.252810] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 02:22:48 (1788589368) [ 1446.789540] Key type .llcrypt unregistered [ 1446.791114] Key type ._llcrypt unregistered [ 1453.126583] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1453.313200] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1453.321826] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1453.337025] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1455.860883] Key type ._llcrypt registered [ 1455.863566] Key type .llcrypt registered [ 1456.133234] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1456.158260] alg: No test for adler32 (adler32-zlib) [ 1457.234745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1457.376262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1457.415514] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1458.326853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1458.342984] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1466.233240] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 02:23:08 (1788589388) [ 1466.508946] LNet: 74397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1466.522303] LNetError: 74397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1466.527497] LNet: Removed LNI 10.1.2.3@tcp [ 1466.554446] LNet: Removed LNI 192.168.203.9@tcp [ 1467.242573] Key type .llcrypt unregistered [ 1467.244885] Key type ._llcrypt unregistered [ 1473.441407] Key type ._llcrypt registered [ 1473.446666] Key type .llcrypt registered [ 1473.738886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1473.768305] alg: No test for adler32 (adler32-zlib) [ 1474.754702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1474.822904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1474.847508] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1475.824587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1475.881082] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1477.556082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.9@tcp [ 1478.624813] LNetError: Unexpected error -22 connecting to 192.168.203.9@tcp at host 192.168.203.9:988 [ 1484.854947] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 02:23:26 (1788589406) [ 1485.160655] LNet: 75526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1485.168490] LNetError: 75526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1485.178546] LNet: Removed LNI 10.1.2.3@tcp [ 1485.201271] LNet: Removed LNI 192.168.203.9@tcp [ 1485.924359] Key type .llcrypt unregistered [ 1485.927530] Key type ._llcrypt unregistered [ 1493.061250] Key type ._llcrypt registered [ 1493.063649] Key type .llcrypt registered [ 1493.260819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1493.290168] alg: No test for adler32 (adler32-zlib) [ 1494.251564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1494.304351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1494.340246] LNet: Added LNI 192.168.203.9@tcp1 [8/256/0/180] [ 1495.272590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1495.309767] LNet: Added LNI 192.168.203.9@tcp2 [8/256/0/180] [ 1496.254943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.9@tcp1 --nid 192.168.203.9@tcp2 [ 1501.935239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.9@tcp1 192.168.203.9@tcp2 [ 1501.969686] LNetError: 76360:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.9@tcp1 to 192.168.203.9@tcp2 - Route table may be misconfigured [ 1501.999378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.9@tcp2 192.168.203.9@tcp1 [ 1509.160745] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 02:23:50 (1788589430) [ 1509.298968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1509.318634] LNet: 76929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1509.327315] LNetError: 76929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1510.367145] LNet: Removed LNI 192.168.203.9@tcp1 [ 1510.374549] LNetError: 76929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1510.396563] LNet: Removed LNI 192.168.203.9@tcp2 [ 1510.439929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1510.504974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1510.523762] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1511.328379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1511.354905] LNet: Added LNI 192.168.203.9@tcp1 [8/256/0/180] [ 1512.200150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 1512.233516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.9@tcp 192.168.203.9@tcp [ 1512.267965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.9@tcp1 192.168.203.9@tcp1 [ 1512.368105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.9@tcp -d 192.168.203.9@tcp -e network_timeout [ 1512.427527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.9@tcp -d 192.168.203.9@tcp1 -e network_timeout [ 1512.498392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp -e network_timeout [ 1512.577437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp1 -e network_timeout [ 1512.648771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 1512.678829] LNet: There was an unexpected network error while writing to 192.168.203.9: rc = -22 [ 1512.706864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1512.858074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1512.981372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1513.581420] LNet: 1 local NIs in recovery (showing 1): 192.168.203.9@tcp [ 1520.196726] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 02:24:01 (1788589441) [ 1520.426408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1520.459963] LNet: 77795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1520.467472] LNetError: 77795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1521.503302] LNet: Removed LNI 192.168.203.9@tcp [ 1522.534465] LNet: Removed LNI 192.168.203.9@tcp1 [ 1522.580654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1522.719756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1523.048068] LNet: 77833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1523.654169] Key type .llcrypt unregistered [ 1523.669424] Key type ._llcrypt unregistered [ 1535.951268] Key type ._llcrypt registered [ 1535.956376] Key type .llcrypt registered [ 1537.479233] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 02:24:19 (1788589459) [ 1537.922340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1537.936966] alg: No test for adler32 (adler32-zlib) [ 1538.855734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1540.390296] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1542.268204] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 02:24:23 (1788589463) [ 1542.746616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1542.797409] LNet: 78930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1542.825146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1542.883966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1542.913871] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1543.843417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1543.858908] LNet: Added LNI 192.168.203.9@tcp1 [8/256/0/180] [ 1544.773986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 1544.816701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp1 [ 1544.856693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp1 [ 1551.179736] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 02:24:33 (1788589473) [ 1552.842505] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg309-server [ 1554.528211] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 02:24:36 (1788589476) [ 1556.297642] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg309-server [ 1557.776153] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 02:24:39 (1788589479) [ 1559.394461] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg309-server [ 1561.106264] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 02:24:43 (1788589483) [ 1562.915360] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg309-server [ 1564.545466] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 02:24:46 (1788589486) [ 1566.651414] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg309-server [ 1568.407806] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 02:24:50 (1788589490) [ 1570.099603] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg309-server [ 1572.068445] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 02:24:53 (1788589493) [ 1574.047197] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg309-server [ 1575.737722] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 02:24:57 (1788589497) [ 1577.681565] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg309-server [ 1579.339850] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 02:25:01 (1788589501) [ 1581.071055] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg309-server [ 1582.623919] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 02:25:04 (1788589504) [ 1582.711262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1582.737835] LNet: 82800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1582.748667] LNetError: 82800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1582.756119] LNet: Removed LNI 192.168.203.9@tcp [ 1583.782394] LNet: Removed LNI 192.168.203.9@tcp1 [ 1583.819147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1583.866897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1583.887817] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1584.530922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1584.607125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 1584.830222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1584.857271] LNet: 82954:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1584.870638] LNetError: 82954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1584.878016] LNetError: 82954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1584.896563] LNet: Removed LNI 192.168.203.9@tcp [ 1584.922525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1584.978597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1585.003675] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1585.862573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 1586.087308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1586.122464] LNetError: 83099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1587.174082] LNet: Removed LNI 192.168.203.9@tcp [ 1587.213135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1587.292462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1587.329612] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1588.183647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1588.285555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 1588.676341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1588.696966] LNet: 83247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1588.704784] LNet: 83247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1588.708839] LNetError: 83247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1589.734284] LNet: Removed LNI 192.168.203.9@tcp [ 1589.747093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1589.779937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1589.795844] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1590.543990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 1590.813577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1591.846418] LNet: Removed LNI 192.168.203.9@tcp [ 1591.872563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1591.933689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1591.953545] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1592.654875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1592.769430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 1592.940774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1592.961168] LNet: 83539:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1592.966900] LNet: 83539:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1592.973413] LNetError: 83539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1592.980599] LNetError: 83539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1592.993499] LNet: Removed LNI 192.168.203.9@tcp [ 1593.008199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1593.048656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1593.069822] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1593.941932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 1594.129316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1595.179178] LNet: Removed LNI 192.168.203.9@tcp [ 1595.229400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1595.313566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1595.349899] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1596.215701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1596.331678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 1596.734107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1597.797988] LNet: Removed LNI 192.168.203.9@tcp [ 1597.869531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1597.978878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1598.049246] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1598.889561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 1599.176284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1600.239932] LNet: Removed LNI 192.168.203.9@tcp [ 1600.310948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1600.448987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1600.498924] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1601.388571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1601.478961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 1601.841449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1601.862513] LNet: 84124:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1601.869288] LNet: 84124:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1601.877382] LNetError: 84124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1601.885038] LNetError: 84124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1602.917380] LNet: Removed LNI 192.168.203.9@tcp [ 1602.980833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1603.079266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1603.114484] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1603.938498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 1604.274408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1605.354667] LNet: Removed LNI 192.168.203.9@tcp [ 1605.394130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1605.474990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1605.498751] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1606.318611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1606.431379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 1606.750877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1607.782380] LNet: Removed LNI 192.168.203.9@tcp [ 1607.820893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1607.904483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1607.927423] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1608.834157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 1609.090192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1610.151641] LNet: Removed LNI 192.168.203.9@tcp [ 1610.179878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1610.272769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1610.295945] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1611.014904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1611.085460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1611.120504] LNet: Removed LNI 192.168.203.9@tcp [ 1611.139420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1611.186331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1611.207258] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1612.001954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1612.077424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1612.110754] LNet: Removed LNI 192.168.203.9@tcp [ 1612.165479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1612.253454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1612.433207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1612.487994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1612.558909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1612.655397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1612.702394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1612.754513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1612.828506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1612.871971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1612.934398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1612.969402] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1613.825504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1613.896008] LNet: Removed LNI 192.168.203.9@tcp [ 1613.928826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1614.005455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1620.566916] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 02:25:42 (1788589542) [ 1620.683445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1620.718975] LNet: 85579:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1620.726915] LNet: 85579:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1620.758364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1620.878244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1620.929779] LNet: Added LNI 192.168.203.9@tcp231 [8/256/0/180] [ 1622.382157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1622.419336] LNetError: 85721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1622.432310] LNetError: 85721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1622.447372] LNet: Removed LNI 192.168.203.9@tcp231 [ 1622.492878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1622.552233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1622.586500] LNet: Added LNI 192.168.203.9@tcp231 [8/256/0/180] [ 1622.603825] LNet: 85729:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1623.243577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1623.286412] LNet: Removed LNI 192.168.203.9@tcp231 [ 1623.306342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1623.361271] LNet: Added LNI 192.168.203.9@tcp231 [8/256/0/180] [ 1624.723659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1624.767825] LNet: Removed LNI 192.168.203.9@tcp231 [ 1624.793811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1624.892412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1624.932862] LNet: Added LNI 192.168.203.9@tcp231 [8/256/0/0] [ 1624.942448] LNet: 85882:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1625.671820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1625.701862] LNet: Removed LNI 192.168.203.9@tcp231 [ 1625.719385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1625.743353] LNet: Added LNI 192.168.203.9@tcp231 [8/256/0/0] [ 1626.944080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1626.979497] LNet: Removed LNI 192.168.203.9@tcp231 [ 1627.009640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1627.106227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1627.128841] LNet: Added LNI 192.168.203.9@tcp231 [8/256/0/60] [ 1627.135970] LNet: 86033:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1627.691495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1627.766546] LNet: Removed LNI 192.168.203.9@tcp231 [ 1627.787934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1627.818176] LNet: Added LNI 192.168.203.9@tcp231 [8/256/0/60] [ 1628.854890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1628.910656] LNet: Removed LNI 192.168.203.9@tcp231 [ 1628.934712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1628.987198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1629.013330] LNet: Added LNI 192.168.203.9@tcp231 [8/256/0/180] [ 1629.452629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1629.488375] LNet: Removed LNI 192.168.203.9@tcp231 [ 1629.516172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1629.555383] LNet: Added LNI 192.168.203.9@tcp231 [8/256/0/180] [ 1637.267768] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 02:25:59 (1788589559) [ 1637.538198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1637.570853] LNet: Removed LNI 192.168.203.9@tcp231 [ 1637.590838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1637.661766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1637.693452] LNet: Added LNI 192.168.203.9@tcp232 [8/256/0/180] [ 1638.364687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1638.418537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1638.449581] LNet: Removed LNI 192.168.203.9@tcp232 [ 1638.484579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1638.517878] LNet: Added LNI 192.168.203.9@tcp232 [8/256/0/180] [ 1639.168816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1639.204657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1639.246823] LNet: Removed LNI 192.168.203.9@tcp232 [ 1639.272785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1639.305927] LNet: Added LNI 192.168.203.9@tcp232 [8/256/0/180] [ 1639.904855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1639.936102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1639.957113] LNet: Removed LNI 192.168.203.9@tcp232 [ 1639.970543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1639.990071] LNet: Added LNI 192.168.203.9@tcp232 [8/256/0/180] [ 1640.525734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1640.576885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1640.638738] LNet: Removed LNI 192.168.203.9@tcp232 [ 1640.651666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1647.637716] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 02:26:09 (1788589569) [ 1649.013494] Key type .llcrypt unregistered [ 1649.021140] Key type ._llcrypt unregistered [ 1656.463126] Key type ._llcrypt registered [ 1656.468692] Key type .llcrypt registered [ 1656.842164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1656.871435] alg: No test for adler32 (adler32-zlib) [ 1658.037319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1658.078095] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1666.249076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.109@tcp [ 1675.717815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1675.760308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1675.787555] LNet: Added LNI 192.168.203.9@tcp1 [8/256/0/180] [ 1677.194863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e local_interrupt -r 2 [ 1677.237282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp1 -m GET -e local_interrupt -r 2 [ 1677.291670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e local_interrupt -r 2 [ 1677.352272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp1 -m GET -e local_interrupt -r 2 [ 1677.384285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp -m GET -e local_interrupt -r 2 [ 1677.420786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp1 -m GET -e local_interrupt -r 2 [ 1677.462483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp -m GET -e local_interrupt -r 2 [ 1677.502715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp1 -m GET -e local_interrupt -r 2 [ 1677.573860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.9@tcp 192.168.203.109@tcp [ 1677.604533] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1677.609770] LNet: 1 local NIs in recovery (showing 1): 192.168.203.9@tcp [ 1677.880244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1677.922154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1677.973781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1678.012000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1678.056979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1678.087942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1678.112218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1678.149396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1678.224498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1678.245847] LNet: 88683:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1678.261553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1678.293601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1678.340523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1678.392688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1678.429987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1678.472461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1678.940539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e local_dropped -r 2 [ 1678.980961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp1 -m GET -e local_dropped -r 2 [ 1679.017995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e local_dropped -r 2 [ 1679.052856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp1 -m GET -e local_dropped -r 2 [ 1679.084605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp -m GET -e local_dropped -r 2 [ 1679.123802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp1 -m GET -e local_dropped -r 2 [ 1679.172862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp -m GET -e local_dropped -r 2 [ 1679.218774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp1 -m GET -e local_dropped -r 2 [ 1679.251815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.9@tcp 192.168.203.109@tcp [ 1679.276729] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1679.535054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1679.581029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1679.607797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1679.649634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1679.684270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1679.722808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1679.786116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1679.844964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1679.898131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1679.922108] LNet: 88806:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1679.940365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1679.973745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1680.014758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1680.058695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1680.095729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1680.132249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1680.752186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e local_aborted -r 2 [ 1680.826715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp1 -m GET -e local_aborted -r 2 [ 1680.898348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e local_aborted -r 2 [ 1680.946651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp1 -m GET -e local_aborted -r 2 [ 1681.032987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp -m GET -e local_aborted -r 2 [ 1681.113679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp1 -m GET -e local_aborted -r 2 [ 1681.171124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp -m GET -e local_aborted -r 2 [ 1681.235329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp1 -m GET -e local_aborted -r 2 [ 1681.267354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.9@tcp 192.168.203.109@tcp [ 1681.297329] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1681.658174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1681.696166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1681.744342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1681.788796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1681.842685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1681.884546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1681.936794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1681.977801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1682.004590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1682.019537] LNet: 88928:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1682.034042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1682.060050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1682.082969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1682.105570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1682.131886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1682.164463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1682.652776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e local_no_route -r 2 [ 1682.701381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp1 -m GET -e local_no_route -r 2 [ 1682.740761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e local_no_route -r 2 [ 1682.796106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp1 -m GET -e local_no_route -r 2 [ 1682.844334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp -m GET -e local_no_route -r 2 [ 1682.907311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp1 -m GET -e local_no_route -r 2 [ 1682.957382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp -m GET -e local_no_route -r 2 [ 1683.009531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp1 -m GET -e local_no_route -r 2 [ 1683.055409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.9@tcp 192.168.203.109@tcp [ 1683.365717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1683.404155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1683.441940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1683.470362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1683.497747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1683.537074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1683.591385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1683.653261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1683.694093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1683.738724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1683.794623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1683.846253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1683.882670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1683.920973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1683.964647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1684.525203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e local_timeout -r 2 [ 1684.617753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp1 -m GET -e local_timeout -r 2 [ 1684.693902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e local_timeout -r 2 [ 1684.769931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp1 -m GET -e local_timeout -r 2 [ 1684.855515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp -m GET -e local_timeout -r 2 [ 1684.915909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp1 -m GET -e local_timeout -r 2 [ 1684.978532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp -m GET -e local_timeout -r 2 [ 1685.034657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp1 -m GET -e local_timeout -r 2 [ 1685.108778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.9@tcp 192.168.203.109@tcp [ 1685.135231] LNet: There was an unexpected network error while writing to 192.168.203.109: rc = -22 [ 1685.139900] LNet: Skipped 1 previous similar message [ 1685.388914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1685.427538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1685.468086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1685.526228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1685.571249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1685.619491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1685.680357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1685.714430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1685.754427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1685.772683] LNet: 89172:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1685.778660] LNet: 89172:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1685.796812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1685.831202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1685.869692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1685.903949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1685.939804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1685.972986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1686.480189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp -m GET -e remote_dropped -r 2 [ 1686.526333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp -d 192.168.203.109@tcp1 -m GET -e remote_dropped -r 2 [ 1686.568443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp -m GET -e remote_dropped -r 2 [ 1686.598476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp -d 192.168.203.9@tcp1 -m GET -e remote_dropped -r 2 [ 1686.632937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp -m GET -e remote_dropped -r 2 [ 1686.674442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.9@tcp1 -d 192.168.203.109@tcp1 -m GET -e remote_dropped -r 2 [ 1686.713687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp -m GET -e remote_dropped -r 2 [ 1686.746608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.9@tcp1 -d 192.168.203.9@tcp1 -m GET -e remote_dropped -r 2 [ 1686.781295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.9@tcp 192.168.203.109@tcp [ 1686.799635] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.109@tcp [ 1687.063714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1687.098570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1687.131456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1687.160537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1687.197429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1687.231668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1687.257296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1687.286673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1687.319710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1687.363891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1687.405828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1687.437555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1687.468869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1687.516879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1687.539756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1692.507237] LNet: 89369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1692.514921] LNetError: 89369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1692.527920] LNet: Removed LNI 192.168.203.9@tcp [ 1692.534824] LNet: Removed LNI 192.168.203.9@tcp1 [ 1692.835793] Key type .llcrypt unregistered [ 1692.840712] Key type ._llcrypt unregistered [ 1707.863139] Key type ._llcrypt registered [ 1707.866564] Key type .llcrypt registered [ 1708.874857] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 02:27:10 (1788589630) [ 1709.453042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1709.466974] alg: No test for adler32 (adler32-zlib) [ 1710.389184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1710.438862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1710.457496] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1711.220625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1711.249313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.9@tcp [ 1711.263688] LNetError: 90345:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.9@tcp to recovery [ 1711.335624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.9@tcp [ 1711.414284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1717.262792] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 02:27:19 (1788589639) [ 1717.417734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1717.471473] LNet: 90929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1717.486071] LNetError: 90929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1717.518466] LNet: Removed LNI 192.168.203.9@tcp [ 1717.558301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1717.624647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1717.647903] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1718.302164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1718.321572] LNet: Added LNI 192.168.203.9@tcp1 [8/256/0/180] [ 1719.404130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.9@tcp [ 1719.433880] LNetError: 91200:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.9@tcp to recovery [ 1719.647370] LNet: 1 local NIs in recovery (showing 1): 192.168.203.9@tcp [ 1719.649520] LNet: There was an unexpected network error while writing to 192.168.203.9: rc = -22 [ 1720.647632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.9@tcp [ 1728.090638] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 02:27:29 (1788589649) [ 1730.091201] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg309-server [ 1731.812509] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 02:27:33 (1788589653) [ 1733.671951] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg309-server [ 1735.446153] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 02:27:37 (1788589657) [ 1737.209772] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg309-server [ 1738.824800] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 02:27:40 (1788589660) [ 1739.110514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1739.138488] LNet: 92823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1739.147189] LNetError: 92823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1739.154083] LNet: Removed LNI 192.168.203.9@tcp [ 1739.173842] LNet: Removed LNI 192.168.203.9@tcp1 [ 1739.199555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1739.290962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1739.338172] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1740.104490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1740.228524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1740.287657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1740.416268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1740.465758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1740.555924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1747.103577] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 02:27:48 (1788589668) [ 1747.259364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1747.285960] LNet: 93554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1747.293630] LNetError: 93554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1747.299061] LNetError: 93554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1747.315326] LNet: Removed LNI 192.168.203.9@tcp [ 1747.329860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1747.371942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1747.389650] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1755.331235] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 02:27:57 (1788589677) [ 1755.489577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1755.520407] LNet: 94252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1755.532410] LNetError: 94252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1755.589820] LNet: Removed LNI 192.168.203.9@tcp [ 1755.668353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1755.739707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1755.778058] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1763.561679] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 02:28:05 (1788589685) [ 1765.159410] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1767.103896] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 02:28:08 (1788589688) [ 1767.734931] LNet: 95293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1767.743489] LNetError: 95293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1767.765387] LNet: Removed LNI 192.168.203.9@tcp [ 1768.516733] Key type .llcrypt unregistered [ 1768.519346] Key type ._llcrypt unregistered [ 1775.119388] Key type ._llcrypt registered [ 1775.126338] Key type .llcrypt registered [ 1775.392886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1775.429608] alg: No test for adler32 (adler32-zlib) [ 1776.420152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1776.459554] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1784.358250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.109@tcp [ 1789.252655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.109@tcp [ 1790.761270] LNet: 95709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1790.767635] LNetError: 95709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1790.788792] LNet: Removed LNI 192.168.203.9@tcp [ 1791.312692] Key type .llcrypt unregistered [ 1791.314400] Key type ._llcrypt unregistered [ 1803.555667] Key type ._llcrypt registered [ 1803.558288] Key type .llcrypt registered [ 1805.447448] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1807.564402] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1809.542696] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 02:28:51 (1788589731) [ 1811.653571] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1813.660925] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 02:28:55 (1788589735) [ 1815.770709] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg309-server [ 1817.613538] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 02:28:59 (1788589739) [ 1819.971547] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg309-server [ 1822.409412] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 02:29:03 (1788589743) [ 1824.427379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1824.486117] alg: No test for adler32 (adler32-zlib) [ 1825.740572] Lustre: Lustre: Build Version: 2.17.58_2_g97ad22e [ 1826.137465] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1827.887241] Key type lgssc registered [ 1829.932896] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1852.235167] Key type lgssc unregistered [ 1852.573584] LNet: 98882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1852.580577] LNetError: 98882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1852.598856] LNet: Removed LNI 192.168.203.9@tcp [ 1853.414488] Key type .llcrypt unregistered [ 1853.416459] Key type ._llcrypt unregistered [ 1868.293492] Key type ._llcrypt registered [ 1868.298394] Key type .llcrypt registered [ 1869.834454] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 02:29:51 (1788589791) [ 1870.168416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1870.191441] alg: No test for adler32 (adler32-zlib) [ 1871.153828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1871.238225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1871.276560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1871.327739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1871.373040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1878.299551] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 02:29:59 (1788589799) [ 1878.604299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1878.637395] LNet: 100295:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1878.684784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1878.770222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1878.794173] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1879.887841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1879.911583] LNet: 100439:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1879.921472] LNetError: 100439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1879.935766] LNet: Removed LNI 192.168.203.9@tcp [ 1879.972730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1880.047079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1880.088229] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1880.785704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1880.829309] LNetError: 100468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1880.841975] LNet: Removed LNI 192.168.203.9@tcp [ 1880.877556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1880.918481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1880.944536] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1888.837756] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 02:30:10 (1788589810) [ 1889.047130] LNet: 101054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1889.059430] LNet: 101054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1889.085125] LNetError: 101054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1889.113601] LNet: Removed LNI 192.168.203.9@tcp [ 1889.836073] Key type .llcrypt unregistered [ 1889.841701] Key type ._llcrypt unregistered [ 1897.086348] Key type ._llcrypt registered [ 1897.089121] Key type .llcrypt registered [ 1897.299363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1897.315557] alg: No test for adler32 (adler32-zlib) [ 1898.271734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1898.363849] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1898.367946] LNet: *** cfs_fail_loc=e003, val=100*** [ 1899.126718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 1899.147879] LNet: *** cfs_fail_loc=e003, val=100*** [ 1899.150642] LNet: Skipped 2 previous similar messages [ 1899.156539] Ping reply received with mapping: [ 1899.158738] found metadata reply NI[192.168.203.9@tcp] [ 1899.161479] 09 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1899.164319] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1899.166949] 5a 5a 5a 5a 5a 5a 5a 5a [ 1899.265368] LNet: 101453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1899.271596] LNetError: 101453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1899.284809] LNet: Removed LNI 192.168.203.9@tcp [ 1899.976734] Key type .llcrypt unregistered [ 1899.984393] Key type ._llcrypt unregistered [ 1911.764082] Key type ._llcrypt registered [ 1911.771990] Key type .llcrypt registered [ 1913.301634] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 02:30:34 (1788589834) [ 1913.955740] Key type .llcrypt unregistered [ 1913.967716] Key type ._llcrypt unregistered [ 1921.606280] Key type ._llcrypt registered [ 1921.609878] Key type .llcrypt registered [ 1921.980737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1922.007282] alg: No test for adler32 (adler32-zlib) [ 1923.081785] LNetError: 102491:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1923.483792] Key type .llcrypt unregistered [ 1923.487382] Key type ._llcrypt unregistered [ 1929.129902] Key type ._llcrypt registered [ 1929.135956] Key type .llcrypt registered [ 1930.522869] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 02:30:52 (1788589852) [ 1931.232168] Key type .llcrypt unregistered [ 1931.239084] Key type ._llcrypt unregistered [ 1938.191321] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1938.200648] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1940.874759] Key type ._llcrypt registered [ 1940.877668] Key type .llcrypt registered [ 1941.142576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1941.164452] alg: No test for adler32 (adler32-zlib) [ 1942.173724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1942.280925] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1953.606980] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 02:31:15 (1788589875) [ 1953.945771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1953.975814] LNet: 104266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1953.983335] LNetError: 104266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1954.011729] LNet: Removed LNI 10.1.2.3@tcp [ 1954.058774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1954.850582] Key type .llcrypt unregistered [ 1954.852352] Key type ._llcrypt unregistered [ 1960.854950] Key type ._llcrypt registered [ 1960.858840] Key type .llcrypt registered [ 1961.009530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1961.030134] alg: No test for adler32 (adler32-zlib) [ 1962.074992] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1962.172404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 1962.362512] LNet: 104551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1962.374477] LNetError: 104551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1962.410465] LNet: Removed LNI 192.168.203.9@tcp [ 1963.140483] Key type .llcrypt unregistered [ 1963.142550] Key type ._llcrypt unregistered [ 1974.907820] Key type ._llcrypt registered [ 1974.909979] Key type .llcrypt registered [ 1976.527050] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 02:31:37 (1788589897) [ 1977.417384] Key type .llcrypt unregistered [ 1977.421230] Key type ._llcrypt unregistered [ 1984.296227] Key type ._llcrypt registered [ 1984.299320] Key type .llcrypt registered [ 1984.525996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1984.555840] alg: No test for adler32 (adler32-zlib) [ 1992.260226] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 02:31:54 (1788589914) [ 1992.645493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1992.690199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1992.784823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1992.832470] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 1993.515722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1993.551846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1993.597728] LNet: Removed LNI 192.168.203.9@tcp [ 1993.828886] LNet: 106292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1994.320203] Key type .llcrypt unregistered [ 1994.331481] Key type ._llcrypt unregistered [ 2006.040491] Key type ._llcrypt registered [ 2006.042762] Key type .llcrypt registered [ 2007.269796] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 02:32:09 (1788589929) [ 2007.880423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2007.897506] alg: No test for adler32 (adler32-zlib) [ 2008.916868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2009.014725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2009.062628] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2010.013956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2010.064829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.9@tcp [ 2010.087792] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2010.090790] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2010.093384] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2010, 0, 0 [ 2010.097659] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2010.102429] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2010.106419] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2010.109957] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2010, 0, 0 [ 2010.114147] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2010.116581] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2010.119856] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2010.122697] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2010, 0, 2060 [ 2010.127499] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2010.130788] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2010.134483] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2010.138149] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2010, 0, 2060 [ 2010.141834] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2016.714442] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 02:32:18 (1788589938) [ 2017.106955] LNet: 107828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2017.117186] LNetError: 107828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2017.136330] LNet: Removed LNI 192.168.203.9@tcp [ 2017.956588] Key type .llcrypt unregistered [ 2017.958808] Key type ._llcrypt unregistered [ 2024.597720] Key type ._llcrypt registered [ 2024.603215] Key type .llcrypt registered [ 2024.807415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2024.826280] alg: No test for adler32 (adler32-zlib) [ 2025.766633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2025.801907] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2033.060804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.109@tcp [ 2043.503878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2043.543894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2043.565734] LNet: Added LNI 192.168.203.9@tcp1 [8/256/0/180] [ 2044.816760] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2044.962170] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2044.982049] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2044.996669] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2047.474463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2047.526977] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2059.827353] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 02:33:01 (1788589981) [ 2060.198416] LNet: 109334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2060.210842] LNetError: 109334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2060.218337] LNet: Removed LNI 192.168.203.9@tcp [ 2061.279505] LNet: Removed LNI 192.168.203.9@tcp1 [ 2061.287322] LNetError: 109334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2061.300880] LNetError: 109334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2061.318587] LNet: Removed LNI 10.1.2.3@tcp99 [ 2062.433792] Key type .llcrypt unregistered [ 2062.436295] Key type ._llcrypt unregistered [ 2070.201153] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2070.204660] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2073.288389] Key type ._llcrypt registered [ 2073.293385] Key type .llcrypt registered [ 2073.806407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2073.890749] alg: No test for adler32 (adler32-zlib) [ 2074.830939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2074.898562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2074.938293] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2075.635814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2075.659716] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2078.799267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2078.908152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2079.027805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2079.044913] LNetError: 110076:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2079.079673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2079.109507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 2079.820173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2079.909889] LNet: 110096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2079.921927] LNetError: 110096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2079.933435] LNet: Removed LNI 10.1.2.3@tcp [ 2081.011654] LNet: Removed LNI 192.168.203.9@tcp [ 2082.296462] Key type .llcrypt unregistered [ 2082.299682] Key type ._llcrypt unregistered [ 2097.615950] Key type ._llcrypt registered [ 2097.623139] Key type .llcrypt registered [ 2098.941434] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 02:33:40 (1788590020) [ 2099.378591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2099.396173] alg: No test for adler32 (adler32-zlib) [ 2100.283534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2100.373300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2100.397110] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2100.997868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2101.128531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg309-client@tcp [ 2109.032541] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 02:33:50 (1788590030) [ 2109.467961] LNet: 111627:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2109.485191] LNetError: 111627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2109.509331] LNet: Removed LNI 192.168.203.9@tcp [ 2110.326420] Key type .llcrypt unregistered [ 2110.339116] Key type ._llcrypt unregistered [ 2118.174587] Key type ._llcrypt registered [ 2118.180138] Key type .llcrypt registered [ 2118.338875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2118.376676] alg: No test for adler32 (adler32-zlib) [ 2119.264711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2119.303217] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2119.330943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2127.853535] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 02:34:09 (1788590049) [ 2130.296416] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2132.647686] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 02:34:13 (1788590053) [ 2134.451889] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2137.334028] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 02:34:18 (1788590058) [ 2137.712782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2137.803357] LNet: 113048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2137.810833] LNetError: 113048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2137.860947] LNet: Removed LNI 192.168.203.9@tcp [ 2137.910350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2138.051112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2138.069157] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2139.269663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2139.322211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2147.647244] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 02:34:29 (1788590069) [ 2147.959651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2147.975741] LNet: 113759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2147.981575] LNetError: 113759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2148.006976] LNet: Removed LNI 192.168.203.9@tcp [ 2148.043646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2148.122826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2148.184399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2148.952475] Key type .llcrypt unregistered [ 2148.959612] Key type ._llcrypt unregistered [ 2161.455921] Key type ._llcrypt registered [ 2161.459197] Key type .llcrypt registered [ 2162.889325] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 02:34:44 (1788590084) [ 2163.396588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2163.416847] alg: No test for adler32 (adler32-zlib) [ 2164.396327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2164.495846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2164.538782] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2165.488806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2165.542177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 2172.298778] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 02:34:54 (1788590094) [ 2172.595808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2172.633722] LNet: 115313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2172.639944] LNetError: 115313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2172.659574] LNet: Removed LNI 192.168.203.9@tcp [ 2172.695205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2172.785378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2172.817392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2181.428534] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 02:35:02 (1788590102) [ 2181.547747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2181.584948] LNet: 115887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2181.626680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2181.772657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2182.392784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2182.434948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2182.523253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2183.056154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2183.082994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2183.150227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2183.168939] LNet: 115927:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2183.197206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2183.312140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2183.910795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2183.967186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2184.047709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2184.560409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2184.594841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2191.737567] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 02:35:13 (1788590113) [ 2191.833401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2191.847962] LNet: 116520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2191.887128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2191.996610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2192.042494] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2192.937766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2192.959195] LNet: Added LNI 192.168.203.9@tcp2 [8/256/0/180] [ 2194.053947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2194.129228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 2194.439042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2194.479724] LNetError: 116807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2194.490430] LNet: Removed LNI 192.168.203.9@tcp [ 2194.512481] LNet: Removed LNI 192.168.203.9@tcp2 [ 2194.549489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2194.617869] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2194.626625] LNet: Added LNI 192.168.203.9@tcp2 [8/256/0/180] [ 2194.633205] LNet: 116813:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2195.424506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 2195.746075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2195.782966] LNetError: 116850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2195.800298] LNetError: 116850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2195.819358] LNet: Removed LNI 192.168.203.9@tcp [ 2195.841745] LNet: Removed LNI 192.168.203.9@tcp2 [ 2195.898508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2195.961772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2196.013543] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2196.946503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2196.992889] LNet: Added LNI 192.168.203.9@tcp2 [8/256/0/180] [ 2197.882090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 2197.920971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2197.975352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 2198.148155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2198.171103] LNet: 117135:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2198.177395] LNet: 117135:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2198.193126] LNetError: 117135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2198.201258] LNetError: 117135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2198.213784] LNet: Removed LNI 192.168.203.9@tcp [ 2199.269794] LNet: Removed LNI 192.168.203.9@tcp2 [ 2199.313857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2199.413648] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2199.422474] LNet: Added LNI 192.168.203.9@tcp2 [8/256/0/180] [ 2199.431461] LNet: 117141:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2200.117599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 2200.327709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2200.346316] LNet: Removed LNI 192.168.203.9@tcp [ 2201.389609] LNet: Removed LNI 192.168.203.9@tcp2 [ 2201.424098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2201.510667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2201.536951] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2202.281177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2202.299255] LNet: Added LNI 192.168.203.9@tcp2 [8/256/0/180] [ 2203.383264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2203.458465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 2203.784459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2203.821567] LNetError: 117458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2203.835203] LNetError: 117458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2203.844286] LNet: Removed LNI 192.168.203.9@tcp [ 2203.866860] LNet: Removed LNI 192.168.203.9@tcp2 [ 2203.910372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2203.998201] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2204.018790] LNet: Added LNI 192.168.203.9@tcp2 [8/256/0/180] [ 2204.025613] LNet: 117464:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2204.688180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 2204.898551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2205.919217] LNet: Removed LNI 192.168.203.9@tcp [ 2205.938156] LNet: Removed LNI 192.168.203.9@tcp2 [ 2205.986547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2206.046445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2206.071069] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2206.927433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2206.978242] LNet: Added LNI 192.168.203.9@tcp2 [8/256/0/180] [ 2208.053917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 2208.095373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2208.162896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 2208.389518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2208.414940] LNet: 117786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2208.420432] LNet: 117786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2209.439136] LNet: Removed LNI 192.168.203.9@tcp [ 2210.472804] LNet: Removed LNI 192.168.203.9@tcp2 [ 2210.564165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2210.654677] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2210.661259] LNet: Added LNI 192.168.203.9@tcp2 [8/256/0/180] [ 2210.666936] LNet: 117792:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2211.422794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 2211.610424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2211.637730] LNet: Removed LNI 192.168.203.9@tcp [ 2212.715897] LNet: Removed LNI 192.168.203.9@tcp2 [ 2212.765678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2212.826784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2212.866520] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2213.595466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2213.615768] LNet: Added LNI 192.168.203.9@tcp2 [8/256/0/180] [ 2214.449468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2214.509907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2214.589800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 2214.964923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2214.990703] LNetError: 118126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2215.007815] LNetError: 118126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2215.019554] LNet: Removed LNI 192.168.203.9@tcp [ 2215.026801] LNet: Removed LNI 192.168.203.9@tcp2 [ 2215.053169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2215.130193] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2215.136196] LNet: Added LNI 192.168.203.9@tcp2 [8/256/0/180] [ 2215.141295] LNet: 118132:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2215.829739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 2216.409871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2217.439233] LNet: Removed LNI 192.168.203.9@tcp [ 2217.453452] LNet: Removed LNI 192.168.203.9@tcp2 [ 2217.512185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2217.582498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2217.617864] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2218.396249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2218.416352] LNet: Added LNI 192.168.203.9@tcp2 [8/256/0/180] [ 2219.441946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 2219.487457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2219.529871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2219.611643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 2220.091667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2221.151157] LNet: Removed LNI 192.168.203.9@tcp [ 2222.194741] LNet: Removed LNI 192.168.203.9@tcp2 [ 2222.238742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2222.379103] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2222.407253] LNet: Added LNI 192.168.203.9@tcp2 [8/256/0/180] [ 2223.250148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 2231.570574] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2233.391808] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 02:35:55 (1788590155) [ 2233.650672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2233.672762] LNet: 119252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2233.678402] LNet: 119252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2233.686131] LNetError: 119252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2233.691153] LNetError: 119252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2233.695570] LNet: Removed LNI 192.168.203.9@tcp [ 2234.730099] LNet: Removed LNI 192.168.203.9@tcp2 [ 2234.760687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2235.056418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2235.108718] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2235.945429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2235.972665] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2236.981599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2236.997779] LNet: Removed LNI 10.1.2.3@tcp [ 2237.010247] LNet: Removed LNI 192.168.203.9@tcp [ 2237.087554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2237.176721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2237.208454] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2237.216293] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2237.473357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2237.485633] LNet: Removed LNI 192.168.203.9@tcp [ 2237.503692] LNet: Removed LNI 10.1.2.3@tcp [ 2244.609343] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 02:36:06 (1788590166) [ 2244.763883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2244.812260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2244.897733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2244.953443] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2246.005470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2246.058858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2246.118755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2246.166792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2246.233684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2246.286970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2246.350226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2246.395621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2246.462488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2246.528911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2246.599490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2246.693852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2246.835963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2255.299935] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 02:36:16 (1788590176) [ 2255.397915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2256.487105] LNet: Removed LNI 192.168.203.9@tcp [ 2256.582438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2256.677344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2256.707621] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2257.494313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2257.524907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2257.564803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2257.613972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2257.687838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2257.716526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2257.740841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2257.773746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2257.796232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2257.823294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2257.845130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2257.874972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2258.015823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2258.074965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2258.100371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2258.124769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2258.147688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2258.201917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2258.278594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2258.345831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2258.401166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2258.469307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2258.523273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.9@tcp [ 2265.993000] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 02:36:27 (1788590187) [ 2266.388913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2266.418257] LNet: 121626:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2266.424949] LNet: 121626:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2266.440814] LNetError: 121626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2266.454794] LNetError: 121626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 2266.478181] LNet: Removed LNI 192.168.203.9@tcp [ 2266.501915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2266.615493] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2266.805363] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2266.817128] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2266.821197] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2269.251653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2269.296526] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2270.151771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2270.184431] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2273.015810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 2273.067345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2273.205225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.9@tcp -m PUT -e local_timeout -r 1 [ 2273.274816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2273.358976] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2273.368712] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2273.707169] LNetError: 121692:0:(lib-move.c:4794:lnet_parse()) 192.168.203.9@tcp, src 192.168.203.9@tcp: Dropping PUT (error -108 looking up sender) [ 2274.723829] LNet: Removed LNI 10.1.2.3@tcp [ 2275.753259] LNet: Removed LNI 192.168.203.9@tcp [ 2276.228238] Key type .llcrypt unregistered [ 2276.231690] Key type ._llcrypt unregistered [ 2290.591918] Key type ._llcrypt registered [ 2290.593735] Key type .llcrypt registered [ 2291.942988] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 02:36:53 (1788590213) [ 2292.377963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2292.405414] alg: No test for adler32 (adler32-zlib) [ 2293.309246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2293.753809] LNet: 123001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2294.401330] Key type .llcrypt unregistered [ 2294.403728] Key type ._llcrypt unregistered [ 2300.503628] Key type ._llcrypt registered [ 2300.509310] Key type .llcrypt registered [ 2300.783285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2300.811242] alg: No test for adler32 (adler32-zlib) [ 2301.794079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2301.827026] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2309.657138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.109@tcp [ 2309.908308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2310.505743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.109@tcp [ 2310.730964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2310.765517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2310.797482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2310.820630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2310.859198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2310.923036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2310.986744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2311.048968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2311.072738] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.109@tcp [ 2311.110761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2311.134602] LNet: 123450:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2311.168154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2311.240362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2311.279683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2311.335199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2311.371724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2311.412190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2312.199918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2312.540307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2312.596407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2312.639162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2312.670259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2312.726926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2312.757851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2312.801384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2312.833517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2312.865137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2312.894613] LNet: 123569:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2312.920521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2312.963871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2313.011599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2313.055276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2313.090247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2313.133829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2313.856426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.109@tcp [ 2314.073288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2314.094882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2314.125572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2314.165769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2314.215552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2314.262663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2314.300356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2314.332268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2314.375348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2314.389233] LNet: 123688:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2314.405262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2314.454663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2314.482850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2314.505754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2314.530082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2314.587494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2315.209831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2315.449669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2315.487941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2315.538492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2315.585470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2315.620275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2315.655284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2315.685478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2315.709831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2315.739431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2315.768655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2315.813045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2315.840706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2315.863990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2315.891734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2315.931694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2316.279469] LNet: 123854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2316.287148] LNetError: 123854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2316.315162] LNet: Removed LNI 192.168.203.9@tcp [ 2316.773608] Key type .llcrypt unregistered [ 2316.777357] Key type ._llcrypt unregistered [ 2333.099849] Key type ._llcrypt registered [ 2333.103917] Key type .llcrypt registered [ 2334.484482] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 02:37:36 (1788590256) [ 2334.923171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2334.943929] alg: No test for adler32 (adler32-zlib) [ 2335.869934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2336.250819] LNet: 124739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2336.928441] Key type .llcrypt unregistered [ 2336.931907] Key type ._llcrypt unregistered [ 2343.717722] Key type ._llcrypt registered [ 2343.720160] Key type .llcrypt registered [ 2343.909504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2343.922506] alg: No test for adler32 (adler32-zlib) [ 2344.867679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2344.914062] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2352.603735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.109@tcp [ 2362.073822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2362.133094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2362.165590] LNet: Added LNI 192.168.203.9@tcp1 [8/256/0/180] [ 2363.491842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2364.140962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.109@tcp [ 2364.441982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2364.452100] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.109@tcp [ 2364.493034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2364.546291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2364.594883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2364.648654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2364.718262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2364.776337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2364.811834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2364.859844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2364.883840] LNet: 125391:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2364.909293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2364.957054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2365.005611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2365.052495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2365.091691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2365.128797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2365.824731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2366.082447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2366.105474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2366.129932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2366.164932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2366.218551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2366.278023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2366.329767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2366.359448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2366.399875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2366.429967] LNet: 125510:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2366.451699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2366.492452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2366.553715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2366.625515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2366.691160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2366.726474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2367.371052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.109@tcp [ 2367.639911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2367.681108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2367.715116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2367.745767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2367.769681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2367.825354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2367.865962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2367.912949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2367.952726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2367.978875] LNet: 125630:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2367.999715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2368.033435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2368.088230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2368.131416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2368.172755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2368.228838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2368.791417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2369.030648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2369.062541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2369.089751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2369.133478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2369.175305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2369.221024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2369.281947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2369.345682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2369.372418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2369.440478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2369.514687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2369.549653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2369.612940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2369.675677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2369.733485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2374.645939] LNet: 125830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2374.650169] LNetError: 125830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2374.654850] LNet: Removed LNI 192.168.203.9@tcp [ 2374.672461] LNet: Removed LNI 192.168.203.9@tcp1 [ 2375.178305] Key type .llcrypt unregistered [ 2375.180055] Key type ._llcrypt unregistered [ 2391.712587] Key type ._llcrypt registered [ 2391.722487] Key type .llcrypt registered [ 2393.089924] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 02:38:34 (1788590314) [ 2393.676447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2393.689931] alg: No test for adler32 (adler32-zlib) [ 2394.637253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2394.696961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2394.743050] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2395.450890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2395.473773] LNet: Added LNI 192.168.203.9@tcp2 [8/256/0/180] [ 2396.301430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.9@tcp [ 2396.488400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2397.599123] LNet: Removed LNI 192.168.203.9@tcp [ 2403.883178] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 02:38:45 (1788590325) [ 2405.567391] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2407.178431] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 02:38:49 (1788590329) [ 2414.698393] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 02:38:56 (1788590336) [ 2422.251897] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 02:39:04 (1788590344) [ 2428.810895] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 02:39:10 (1788590350) [ 2436.759176] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 02:39:18 (1788590358) [ 2446.171322] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 02:39:27 (1788590367) [ 2453.613993] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 02:39:35 (1788590375) [ 2460.834555] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 02:39:42 (1788590382) [ 2468.048581] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 02:39:49 (1788590389) [ 2474.663587] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 02:39:56 (1788590396) [ 2480.817486] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 02:40:02 (1788590402) [ 2487.897497] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 02:40:09 (1788590409) [ 2494.863266] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 02:40:16 (1788590416) [ 2500.742628] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 02:40:22 (1788590422) [ 2506.730573] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 02:40:28 (1788590428) [ 2513.310967] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 02:40:35 (1788590435) [ 2520.783653] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 02:40:42 (1788590442) [ 2527.282950] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 02:40:49 (1788590449) [ 2527.500597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2527.532428] LNet: 137523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2527.542560] LNetError: 137523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2527.567203] LNet: Removed LNI 192.168.203.9@tcp2 [ 2527.601248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2527.733556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2527.762884] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2529.019686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2529.050650] LNet: 137665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2529.056173] LNetError: 137665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2529.074800] LNet: Removed LNI 192.168.203.9@tcp [ 2529.091380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2529.145761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2529.168222] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2535.450284] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 02:40:57 (1788590457) [ 2542.431837] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 02:41:04 (1788590464) [ 2549.130825] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 02:41:11 (1788590471) [ 2556.153460] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 02:41:17 (1788590477) [ 2562.305519] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 02:41:24 (1788590484) [ 2569.740633] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 02:41:31 (1788590491) [ 2576.381610] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 02:41:38 (1788590498) [ 2584.753634] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 02:41:46 (1788590506) [ 2593.501088] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 02:41:54 (1788590514) [ 2601.651343] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 02:42:03 (1788590523) [ 2607.321332] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 02:42:09 (1788590529) [ 2613.132280] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 02:42:15 (1788590535) [ 2620.437973] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2401 sec ========== 02:42:22 (1788590542) [ 2620.827225] LNet: 145137:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2620.834632] LNetError: 145137:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2620.850198] LNet: Removed LNI 192.168.203.9@tcp [ 2622.077569] Key type .llcrypt unregistered [ 2622.079276] Key type ._llcrypt unregistered [ 2634.973315] Key type ._llcrypt registered [ 2634.975832] Key type .llcrypt registered [ 2635.444089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2635.473101] alg: No test for adler32 (adler32-zlib) [ 2636.464494] Lustre: Lustre: Build Version: 2.17.58_2_g97ad22e [ 2636.686261] LNet: Added LNI 192.168.203.9@tcp [8/256/0/180] [ 2638.335167] Key type lgssc registered [ 2639.411492] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2696.688621] Lustre: Mounted lustre-client - version 2.17.58_2_g97ad22e [ 2701.712100] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2711.590542] Lustre: Unmounted lustre-client [ 2742.799894] Key type lgssc unregistered [ 2743.064846] LNet: 148089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2743.074863] LNetError: 148089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2743.096946] LNet: Removed LNI 192.168.203.9@tcp [ 2743.917336] Key type .llcrypt unregistered [ 2743.921577] Key type ._llcrypt unregistered