[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 430669269 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002306] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.006881] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.009150] LSM: Security Framework initializing [ 0.010100] Yama: becoming mindful. [ 0.011040] SELinux: Initializing. [ 0.012068] *** VALIDATE selinux *** [ 0.020688] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025451] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026150] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027116] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028102] *** VALIDATE tmpfs *** [ 0.030259] *** VALIDATE proc *** [ 0.031254] *** VALIDATE cgroup *** [ 0.032011] *** VALIDATE cgroup2 *** [ 0.033104] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034150] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036031] Spectre V2 : User space: Vulnerable [ 0.037009] Speculative Store Bypass: Vulnerable [ 0.040252] debug: unmapping init [mem 0xffffffffaee59000-0xffffffffaee60fff] [ 0.042899] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043665] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044024] ... version: 2 [ 0.045010] ... bit width: 48 [ 0.046024] ... generic registers: 4 [ 0.047011] ... value mask: 0000ffffffffffff [ 0.048013] ... max period: 00007fffffffffff [ 0.049015] ... fixed-purpose events: 3 [ 0.050011] ... event mask: 000000070000000f [ 0.052221] rcu: Hierarchical SRCU implementation. [ 0.054374] smp: Bringing up secondary CPUs ... [ 0.055559] x86: Booting SMP configuration: [ 0.056023] .... node #0, CPUs: #1 #2 #3 [ 0.059611] smp: Brought up 1 node, 4 CPUs [ 0.061015] smpboot: Max logical packages: 1 [ 0.062019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.264026] node 0 deferred pages initialised in 200ms [ 0.267118] devtmpfs: initialized [ 0.268248] x86/mm: Memory block size: 128MB [ 0.272032] gcov: version magic: 0x41383552 [ 0.275360] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.278113] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.281439] pinctrl core: initialized pinctrl subsystem [ 0.284244] [ 0.284869] ************************************************************* [ 0.288019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.290016] ** ** [ 0.293016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.296019] ** ** [ 0.299029] ** This means that this kernel is built to expose internal ** [ 0.301013] ** IOMMU data structures, which may compromise security on ** [ 0.303018] ** your system. ** [ 0.306018] ** ** [ 0.309019] ** If you see this message and you are not debugging the ** [ 0.312021] ** kernel, report this immediately to your vendor! ** [ 0.315019] ** ** [ 0.318020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.321022] ************************************************************* [ 0.323823] NET: Registered protocol family 16 [ 0.325369] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.328086] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.331082] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.335134] cpuidle: using governor menu [ 0.336714] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.339673] PCI: Using configuration type 1 for base access [ 0.342145] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.351123] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.353062] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.357122] cryptd: max_cpu_qlen set to 1000 [ 0.361313] ACPI: Added _OSI(Module Device) [ 0.363020] ACPI: Added _OSI(Processor Device) [ 0.365015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.367014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.372879] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.379316] ACPI: Interpreter enabled [ 0.381067] ACPI: PM: (supports S0 S3 S4 S5) [ 0.382013] ACPI: Using IOAPIC for interrupt routing [ 0.384164] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.388446] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.398000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.400038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.402017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.405079] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.411389] acpiphp: Slot [2] registered [ 0.413139] acpiphp: Slot [5] registered [ 0.414125] acpiphp: Slot [6] registered [ 0.416133] acpiphp: Slot [3] registered [ 0.417211] acpiphp: Slot [4] registered [ 0.419168] acpiphp: Slot [7] registered [ 0.421130] acpiphp: Slot [8] registered [ 0.423190] acpiphp: Slot [9] registered [ 0.425155] acpiphp: Slot [10] registered [ 0.427115] acpiphp: Slot [11] registered [ 0.428166] acpiphp: Slot [12] registered [ 0.430143] acpiphp: Slot [13] registered [ 0.432292] acpiphp: Slot [14] registered [ 0.433159] acpiphp: Slot [15] registered [ 0.435161] acpiphp: Slot [16] registered [ 0.437141] acpiphp: Slot [17] registered [ 0.438145] acpiphp: Slot [18] registered [ 0.440166] acpiphp: Slot [19] registered [ 0.442150] acpiphp: Slot [20] registered [ 0.443187] acpiphp: Slot [21] registered [ 0.445156] acpiphp: Slot [22] registered [ 0.447144] acpiphp: Slot [23] registered [ 0.449104] acpiphp: Slot [24] registered [ 0.450148] acpiphp: Slot [25] registered [ 0.452189] acpiphp: Slot [26] registered [ 0.453129] acpiphp: Slot [27] registered [ 0.455153] acpiphp: Slot [28] registered [ 0.456172] acpiphp: Slot [29] registered [ 0.458147] acpiphp: Slot [30] registered [ 0.460138] acpiphp: Slot [31] registered [ 0.461132] PCI host bridge to bus 0000:00 [ 0.463029] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.465037] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.468036] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.471037] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.474053] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.477032] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.480189] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.483031] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.488416] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.496538] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.501059] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.506026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.509031] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.512020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.515560] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.520802] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.523059] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.526905] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.532021] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.542021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.547021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.554165] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.560014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.568016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.585038] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.595835] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.603022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.608021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.626025] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.635760] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.639431] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.642414] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.644415] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.646212] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.653034] iommu: Default domain type: Passthrough [ 0.654395] SCSI subsystem initialized [ 0.655104] ACPI: bus type USB registered [ 0.656125] usbcore: registered new interface driver usbfs [ 0.658197] usbcore: registered new interface driver hub [ 0.661125] usbcore: registered new device driver usb [ 0.663250] pps_core: LinuxPPS API ver. 1 registered [ 0.665017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.669090] PTP clock support registered [ 0.672134] EDAC MC: Ver: 3.0.0 [ 0.674175] PCI: Using ACPI for IRQ routing [ 0.676135] NetLabel: Initializing [ 0.677013] NetLabel: domain hash size = 128 [ 0.679018] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.681115] NetLabel: unlabeled traffic allowed by default [ 0.684166] vgaarb: loaded [ 0.686312] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.688022] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.694360] clocksource: Switched to clocksource kvm-clock [ 0.801928] VFS: Disk quotas dquot_6.6.0 [ 0.803726] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.806776] *** VALIDATE ramfs *** [ 0.808530] *** VALIDATE hugetlbfs *** [ 0.811491] pnp: PnP ACPI init [ 0.814479] pnp: PnP ACPI: found 6 devices [ 0.833236] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.836750] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.839025] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.841400] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.843985] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.846458] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.849426] NET: Registered protocol family 2 [ 0.852153] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.856891] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.860379] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.865632] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.868905] TCP: Hash tables configured (established 65536 bind 65536) [ 0.871910] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.874979] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.877827] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.880769] NET: Registered protocol family 1 [ 0.883311] RPC: Registered named UNIX socket transport module. [ 0.885772] RPC: Registered udp transport module. [ 0.888370] RPC: Registered tcp transport module. [ 0.890378] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.893021] NET: Registered protocol family 44 [ 0.894910] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.897378] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.899761] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.902416] PCI: CLS 0 bytes, default 64 [ 0.904296] Unpacking initramfs... [ 2.317964] debug: unmapping init [mem 0xffffa03d3cc64000-0xffffa03d3ffcffff] [ 2.323026] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.325854] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.329325] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.825517] Initialise system trusted keyrings [ 2.827413] Key type blacklist registered [ 2.829675] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.838733] zbud: loaded [ 2.842166] *** VALIDATE nfs *** [ 2.843786] *** VALIDATE nfs4 *** [ 2.845151] pstore: using deflate compression [ 2.848173] Platform Keyring initialized [ 2.954591] NET: Registered protocol family 38 [ 2.956773] Key type asymmetric registered [ 2.958820] Asymmetric key parser 'x509' registered [ 2.961050] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.964195] io scheduler mq-deadline registered [ 2.965936] io scheduler kyber registered [ 2.968149] io scheduler bfq registered [ 2.970346] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.973718] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.977456] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.980879] ACPI: Power Button [PWRF] [ 2.987298] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.994060] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.005200] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.034146] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.067100] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.072744] Non-volatile memory driver v1.3 [ 3.074460] Linux agpgart interface v0.103 [ 3.105246] virtio_blk virtio1: [vda] 149768 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.108235] vda: detected capacity change from 0 to 76681216 [ 3.122929] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.126228] vdb: detected capacity change from 0 to 1073741824 [ 3.133868] libphy: Fixed MDIO Bus: probed [ 3.140753] usbcore: registered new interface driver usbserial_generic [ 3.143492] usbserial: USB Serial support registered for generic [ 3.145901] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.150135] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.151608] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.154390] mousedev: PS/2 mouse device common for all mice [ 3.156825] rtc_cmos 00:05: RTC can wake from S4 [ 3.159394] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.161801] rtc_cmos 00:05: registered as rtc0 [ 3.165161] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.165879] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.171357] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.171930] intel_pstate: CPU model not supported [ 3.178385] hid: raw HID events driver (C) Jiri Kosina [ 3.180818] usbcore: registered new interface driver usbhid [ 3.182644] usbhid: USB HID core driver [ 3.184348] drop_monitor: Initializing network drop monitor service [ 3.187184] Initializing XFRM netlink socket [ 3.189472] NET: Registered protocol family 10 [ 3.192381] Segment Routing with IPv6 [ 3.193969] NET: Registered protocol family 17 [ 3.196404] mpls_gso: MPLS GSO support [ 3.201966] RAS: Correctable Errors collector initialized. [ 3.204249] AVX version of gcm_enc/dec engaged. [ 3.205910] AES CTR mode by8 optimization enabled [ 3.288702] sched_clock: Marking stable (3288598875, 0)->(4206660102, -918061227) [ 3.292246] registered taskstats version 1 [ 3.294247] Loading compiled-in X.509 certificates [ 3.297195] zswap: loaded using pool lzo/zbud [ 3.344177] Key type big_key registered [ 3.356910] Key type encrypted registered [ 3.358912] ima: No TPM chip found, activating TPM-bypass! [ 3.361448] ima: Allocated hash algorithm: sha1 [ 3.363560] ima: No architecture policies found [ 3.365569] evm: Initialising EVM extended attributes: [ 3.367958] evm: security.selinux [ 3.369337] evm: security.ima [ 3.370556] evm: security.capability [ 3.371769] evm: HMAC attrs: 0x1 [ 3.374260] rtc_cmos 00:05: setting system clock to 2026-09-05 21:10:11 UTC (1788642611) [ 3.380336] debug: unmapping init [mem 0xffffffffafe03000-0xffffffffafffffff] [ 3.383638] debug: unmapping init [mem 0xffffffffaeb82000-0xffffffffaee58fff] [ 3.392127] Write protecting the kernel read-only data: 28672k [ 3.395579] debug: unmapping init [mem 0xffffffffad203000-0xffffffffad3fffff] [ 3.397559] debug: unmapping init [mem 0xffffffffadb14000-0xffffffffadbfffff] [ 3.436923] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.444822] systemd[1]: Detected virtualization kvm. [ 3.446819] systemd[1]: Detected architecture x86-64. [ 3.448294] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.475477] systemd[1]: No hostname configured. [ 3.477086] systemd[1]: Set hostname to . [ 3.478744] random: systemd: uninitialized urandom read (16 bytes read) [ 3.482457] systemd[1]: Initializing machine ID from random generator. [ 3.522408] random: ln: uninitialized urandom read (6 bytes read) [ 3.610365] random: systemd: uninitialized urandom read (16 bytes read) [ 3.614267] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.622784] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.628787] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.315909] device-mapper: uevent: version 1.0.3 [ 4.318707] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. [ 4.650130] random: fast init done Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.117603] virtio_net virtio0 ens2: renamed from eth0 [ 5.242874] scsi host0: ata_piix [ 5.296226] scsi host1: ata_piix [ 5.299289] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.302339] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.604563] dracut-initqueue[588]: RTNETLINK answers: File exists [ 9.876963] random: crng init done [ 9.878774] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.350676] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.536860] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.801494] SELinux: Disabled at runtime. [ 11.860792] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.867835] systemd[1]: Detected virtualization kvm. [ 11.869456] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.398219] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.401298] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.406083] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.409461] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.412304] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.420977] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.442321] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ 12.634106] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.923310] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.209437] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.320520] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.376483] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.392150] EDAC sbridge: Ver: 1.1.2 [ 14.761242] Key type dns_resolver registered [ 15.061718] NFS: Registering the id_resolver key type [ 15.063757] Key type id_resolver registered [ 15.065311] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg336-client login: [ 56.858053] libcfs: loading out-of-tree module taints kernel. [ 57.083093] Key type ._llcrypt registered [ 57.087560] Key type .llcrypt registered [ 57.740606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 57.757304] alg: No test for adler32 (adler32-zlib) [ 59.236379] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 60.020070] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 61.743267] Key type lgssc registered [ 63.692791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 89.091065] hrtimer: interrupt took 2499030 ns [ 190.894236] Lustre: Mounted lustre-client - version 2.17.58_3_ge47db3b [ 195.735929] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 207.910541] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing check_logdir /tmp/testlogs/ [ 213.086952] Lustre: DEBUG MARKER: oleg336-client.virtnet: executing yml_node [ 216.543385] Lustre: lustre-OST0000-osc-ffffa03d885fc800: disconnect after 23s idle [ 217.198922] Lustre: DEBUG MARKER: Client: 2.17.58.3 [ 219.444328] Lustre: DEBUG MARKER: MDS: 2.17.58.3 [ 221.834565] Lustre: DEBUG MARKER: OSS: 2.17.58.3 [ 223.300206] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sat Sep 5 17:13:50 EDT 2026 [ 237.713538] Lustre: Unmounted lustre-client [ 264.944537] Key type lgssc unregistered [ 265.308659] LNet: 8401:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 265.317639] LNetError: 8401:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 265.343549] LNet: Removed LNI 192.168.203.36@tcp [ 266.275226] Key type .llcrypt unregistered [ 266.277507] Key type ._llcrypt unregistered [ 291.794487] Key type ._llcrypt registered [ 291.796517] Key type .llcrypt registered [ 292.030908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 292.044608] alg: No test for adler32 (adler32-zlib) [ 293.080146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 293.181985] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 293.217716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 297.610500] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 297.802376] LNet: 9280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 297.807924] LNetError: 9280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 297.828784] LNet: Removed LNI 192.168.203.36@tcp [ 298.489413] Key type .llcrypt unregistered [ 298.492805] Key type ._llcrypt unregistered [ 305.664101] Key type ._llcrypt registered [ 305.668727] Key type .llcrypt registered [ 307.198059] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 17:15:13 (1788642913) [ 307.496728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 307.509652] alg: No test for adler32 (adler32-zlib) [ 308.505162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 308.595866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 308.628156] LNet: 9687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 308.663385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 308.725073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 308.739820] LNet: 9694:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 315.189865] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 17:15:21 (1788642921) [ 315.302059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 315.362806] LNet: 10266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 315.401916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 315.540644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 322.657102] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 17:15:29 (1788642929) [ 322.739574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 322.764941] LNet: 10851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 322.800629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 322.939678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 330.415589] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 17:15:36 (1788642936) [ 330.533924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.565975] LNet: 11434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 330.593772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.681345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 337.501538] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 17:15:44 (1788642944) [ 337.601630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 337.629552] LNet: 12022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 337.661851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 337.775194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 338.260239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 345.197567] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 17:15:51 (1788642951) [ 345.322811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 345.397695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.573386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 346.426491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 347.138513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 354.533728] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 17:16:01 (1788642961) [ 354.653937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 354.703206] LNet: 13210:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 354.722725] LNet: 13210:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 354.760244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 354.880984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 362.232199] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 17:16:08 (1788642968) [ 362.353821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 362.431105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 362.528386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 362.581322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 363.254265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 363.320465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 363.916435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 363.970079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 364.716112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 364.765231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 365.438232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 365.505945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 366.032310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 366.073798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 366.514366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 366.564897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 367.103911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 367.146715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 367.680302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 367.724829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 368.221555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 368.248949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 368.726738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 368.780594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 369.323353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 369.400809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 369.908895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 369.944581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 377.775760] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 17:16:24 (1788642984) [ 377.922778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 377.953754] LNet: 14476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 377.958873] LNet: 14476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 377.983809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 378.088609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 378.122146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 385.895918] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 17:16:32 (1788642992) [ 386.326034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 386.383367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 386.469716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 386.497334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 394.038253] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 17:16:40 (1788643000) [ 394.276586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 394.337190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.420473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 394.460076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 402.124671] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 17:16:48 (1788643008) [ 402.217750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 402.305580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 402.401548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 402.476460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 411.419855] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 17:16:57 (1788643017) [ 411.522233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 411.569667] LNet: 16822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 411.587758] LNet: 16822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 411.638466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 411.783736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 411.832955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 418.984796] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 17:17:05 (1788643025) [ 419.220367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 419.286931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 419.407366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 419.465159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 427.400987] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 17:17:13 (1788643033) [ 427.592842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 427.658636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 427.813052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 428.692483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 436.505877] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 17:17:22 (1788643042) [ 436.594096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 436.673775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 436.953756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 437.313688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 445.789080] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 17:17:32 (1788643052) [ 446.108521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.187601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.362610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 446.636260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 453.196906] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 17:17:40 (1788643060) [ 453.620733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.667930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.082353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 454.385076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 460.914493] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 17:17:47 (1788643067) [ 461.270382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 461.313893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.402542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 461.454650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 468.745639] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 17:17:55 (1788643075) [ 469.010978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 469.089098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 469.188480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 469.259978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 476.470664] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 17:18:03 (1788643083) [ 476.758071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.792522] LNet: 21685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 476.796037] LNet: 21685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 476.820579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.916597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 485.414385] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 17:18:11 (1788643091) [ 485.590665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.692233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.865880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 494.806976] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 17:18:20 (1788643100) [ 494.956092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 495.023940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 495.153818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 495.925959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 504.793836] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 17:18:31 (1788643111) [ 504.954881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.007571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.136463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 505.193952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 513.970273] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 17:18:40 (1788643120) [ 514.125397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 514.229947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 514.344896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 514.388709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 523.238677] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 17:18:49 (1788643129) [ 523.380725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 523.455620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 523.632192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 523.709506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 531.446745] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 17:18:57 (1788643137) [ 531.894155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 531.970674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 532.077855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 532.164899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 539.621596] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 17:19:06 (1788643146) [ 539.846850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 539.953869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.013995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 540.110973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 540.141097] LNetError: 25803:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 540.210780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 540.270849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 546.516553] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 17:19:13 (1788643153) [ 546.642953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 546.710856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.843346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.KmglgJHb5h [ 546.941766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.KmglgJHb5h [ 547.036041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.KmglgJHb5h [ 553.787117] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 17:19:20 (1788643160) [ 553.872439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 553.930980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 554.050624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 554.102381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 554.186611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 560.491513] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 17:19:27 (1788643167) [ 560.605431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 560.678304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 560.803320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 560.858250] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 562.752858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.36@tcp [ 564.404734] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 566.495149] Key type lgssc registered [ 567.742136] Lustre: Echo OBD driver; http://www.lustre.org/ [ 571.412320] LNetError: 27563:0:(lib-move.c:2169:lnet_handle_find_routed_path()) peer 192.168.203.36@tcp99 has no available nets [ 574.147995] Key type lgssc unregistered [ 574.536429] LNetError: 28670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 574.564227] LNet: Removed LNI 192.168.203.36@tcp [ 575.182476] Key type .llcrypt unregistered [ 575.189460] Key type ._llcrypt unregistered [ 582.365212] Key type ._llcrypt registered [ 582.368929] Key type .llcrypt registered [ 583.931833] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 17:19:50 (1788643190) [ 584.564486] Key type .llcrypt unregistered [ 584.570798] Key type ._llcrypt unregistered [ 591.154511] Key type ._llcrypt registered [ 591.156784] Key type .llcrypt registered [ 591.327496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 591.343739] alg: No test for adler32 (adler32-zlib) [ 592.285322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.345827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 592.367234] LNet: 29580:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 592.375868] LNet: 29580:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 592.385382] LNet: 29580:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 592.412962] LNet: Message forwarding will be enabled in 2s [ 595.487233] LNet: Message forwarding enabled [ 595.810425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 595.838380] LNet: Message forwarding will stop in 2s to 6s [ 599.599797] LNet: Message forwarding disabled [ 600.384836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 600.408782] LNet: 29678:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 600.417640] LNet: 29678:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 600.432882] LNet: 29678:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 600.465950] LNet: Message forwarding will be enabled in 2s [ 600.700395] LNet: 29691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 601.382973] Key type .llcrypt unregistered [ 601.385819] Key type ._llcrypt unregistered [ 612.726615] Key type ._llcrypt registered [ 612.729505] Key type .llcrypt registered [ 614.367741] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 17:20:20 (1788643220) [ 614.731981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 614.768302] alg: No test for adler32 (adler32-zlib) [ 615.758667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.858893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 621.565391] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 17:20:28 (1788643228) [ 621.711199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 621.751258] LNet: 31091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 621.767569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.815715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 621.855869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 621.885252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 621.912171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 621.942245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 621.970906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 622.000870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 622.030223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 622.064741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 622.101585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 622.136373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 622.184950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 622.260248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 622.313518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 622.369273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 622.404946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 622.466647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 622.524780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 622.579377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 622.625838] LNetError: 31134:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 629.287911] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 17:20:36 (1788643236) [ 629.561472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 629.605702] LNet: 31705:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 629.626359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 629.717992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 637.094274] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 17:20:43 (1788643243) [ 638.586842] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg336-server [ 640.510778] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 17:20:47 (1788643247) [ 640.652499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 640.690583] LNet: 32630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 640.741381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 640.848707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 640.904214] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 641.757182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[37-41]@tcp [ 641.786976] LNet: 32769:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 641.862077] LNetError: 32639:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.37@tcp is being used as a gateway but routing feature is not turned on [ 641.892652] LNetError: 32639:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.37@tcp has gone from up to down [ 647.673604] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 17:20:54 (1788643254) [ 649.333790] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg336-server [ 651.246759] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 17:20:57 (1788643257) [ 651.502638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 651.535493] LNet: 33705:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 651.546959] LNetError: 33705:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 651.572491] LNet: Removed LNI 192.168.203.36@tcp [ 651.595597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 651.639655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 651.666088] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 652.359307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[37-42/2]@tcp [ 652.385585] LNet: 33822:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 652.405707] LNet: 33822:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 652.435492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[37-42/2]@tcp [ 652.436732] LNetError: 33709:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.41@tcp is being used as a gateway but routing feature is not turned on [ 652.468598] LNetError: 33709:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 652.485638] LNetError: 33709:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.41@tcp has gone from up to down [ 652.495695] LNetError: 33709:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 658.361735] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 17:21:05 (1788643265) [ 658.622965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 658.647841] LNet: 34414:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 658.669875] LNetError: 34414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 658.701933] LNet: Removed LNI 192.168.203.36@tcp [ 658.724608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 658.813784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 658.907837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 658.980756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.087807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 659.164141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 659.213446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.247735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 659.309727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 659.346423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.394659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 659.471333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 659.512628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.580157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 659.685986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 659.720910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.802219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 659.904603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 659.935624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.980642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 660.047782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 660.075987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 660.138501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 660.258653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 660.301902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 660.345643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 660.419891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 660.467888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 660.543217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 660.607432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 660.648876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 660.723764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 660.788726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 660.828855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 660.902938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 667.634733] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 17:21:14 (1788643274) [ 669.955727] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg336-server [ 671.360581] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 17:21:18 (1788643278) [ 673.254457] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg336-server [ 675.181356] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 17:21:21 (1788643281) [ 675.420549] LNet: 35817:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 675.433611] LNet: 35817:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 676.185209] Key type .llcrypt unregistered [ 676.188511] Key type ._llcrypt unregistered [ 682.699174] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 682.866857] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 682.871719] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 682.874753] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 685.575143] Key type ._llcrypt registered [ 685.577891] Key type .llcrypt registered [ 685.831770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 685.848973] alg: No test for adler32 (adler32-zlib) [ 686.818617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.877909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 686.902471] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 687.634515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 687.661990] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 689.930792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 689.953972] LNet: Removed LNI 10.1.2.3@tcp [ 690.125976] LNet: 36535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 690.133699] LNetError: 36535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 690.176717] LNet: Removed LNI 192.168.203.36@tcp [ 690.848395] Key type .llcrypt unregistered [ 690.857578] Key type ._llcrypt unregistered [ 701.379430] Key type ._llcrypt registered [ 701.384853] Key type .llcrypt registered [ 703.035919] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 17:21:49 (1788643309) [ 703.703068] Key type .llcrypt unregistered [ 703.708146] Key type ._llcrypt unregistered [ 709.786288] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 709.903253] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 709.906953] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 709.914839] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 712.476197] Key type ._llcrypt registered [ 712.480356] Key type .llcrypt registered [ 712.663847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 712.681449] alg: No test for adler32 (adler32-zlib) [ 713.626693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.726565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 713.763799] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 714.448667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 714.465596] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 717.987543] LNet: 38025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 717.998786] LNetError: 38025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 718.009481] LNet: Removed LNI 10.1.2.3@tcp [ 718.030310] LNet: Removed LNI 192.168.203.36@tcp [ 718.877183] Key type .llcrypt unregistered [ 718.880474] Key type ._llcrypt unregistered [ 730.035522] Key type ._llcrypt registered [ 730.042290] Key type .llcrypt registered [ 731.543626] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 17:22:17 (1788643337) [ 732.012521] Key type .llcrypt unregistered [ 732.014986] Key type ._llcrypt unregistered [ 737.551827] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 737.654937] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 737.660305] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 737.672201] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 740.457766] Key type ._llcrypt registered [ 740.464326] Key type .llcrypt registered [ 740.756611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 740.776549] alg: No test for adler32 (adler32-zlib) [ 741.729834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 741.784533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 741.816092] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 743.757751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 743.776377] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 745.496974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 745.516630] LNet: Removed LNI 10.1.2.3@tcp [ 745.542396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 745.572212] LNet: Removed LNI 10.1.2.31@tcp [ 745.589765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 745.635195] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 746.241581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 746.271429] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 747.117577] LNet: 39941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 747.130966] LNetError: 39941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 747.141219] LNet: Removed LNI 10.1.2.31@tcp [ 747.164771] LNet: Removed LNI 10.1.2.3@tcp [ 747.682684] Key type .llcrypt unregistered [ 747.690111] Key type ._llcrypt unregistered [ 758.651144] Key type ._llcrypt registered [ 758.654482] Key type .llcrypt registered [ 759.882655] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 17:22:46 (1788643366) [ 760.472680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 760.502074] alg: No test for adler32 (adler32-zlib) [ 761.503513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.572830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 761.604271] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 762.366173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 762.398341] LNet: 40892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 762.407798] LNetError: 40892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 762.430406] LNet: Removed LNI 192.168.203.36@tcp [ 762.461576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 762.542141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.36@tcp [ 762.569207] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 763.608635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.36@tcp [ 763.658560] LNet: Removed LNI 192.168.203.36@tcp [ 763.763214] LNet: 41027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 764.512777] Key type .llcrypt unregistered [ 764.516362] Key type ._llcrypt unregistered [ 776.240883] Key type ._llcrypt registered [ 776.246272] Key type .llcrypt registered [ 777.508765] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 17:23:04 (1788643384) [ 779.017800] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg336-server [ 780.674616] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 17:23:07 (1788643387) [ 781.248719] Key type .llcrypt unregistered [ 781.255616] Key type ._llcrypt unregistered [ 786.899223] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 787.164036] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 787.167839] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 787.170566] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 789.693179] Key type ._llcrypt registered [ 789.695108] Key type .llcrypt registered [ 789.842856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 789.859979] alg: No test for adler32 (adler32-zlib) [ 790.773632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 790.843267] LNet: Added LNI 192.168.203.36@tcp1 [8/256/0/180] [ 791.577373] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 793.446949] LNet: 42858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 793.453760] LNetError: 42858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 793.460671] LNet: Removed LNI 192.168.203.36@tcp1 [ 793.479575] LNet: Removed LNI 10.1.2.3@tcp2 [ 794.020545] Key type .llcrypt unregistered [ 794.023852] Key type ._llcrypt unregistered [ 804.942360] Key type ._llcrypt registered [ 804.947451] Key type .llcrypt registered [ 806.538173] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 17:23:32 (1788643412) [ 806.939651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 806.969165] alg: No test for adler32 (adler32-zlib) [ 807.896642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 807.951373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 807.983414] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 808.717941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 808.734931] LNet: 43825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 808.743722] LNetError: 43825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 808.760860] LNet: Removed LNI 192.168.203.36@tcp [ 808.777480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 808.834981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 808.852328] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 815.670924] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 17:23:42 (1788643422) [ 815.778592] LNet: 44409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.787145] LNetError: 44409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 815.819791] LNet: Removed LNI 192.168.203.36@tcp [ 816.509547] Key type .llcrypt unregistered [ 816.516954] Key type ._llcrypt unregistered [ 822.297801] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 822.456818] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 822.461947] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 822.470237] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 825.116622] Key type ._llcrypt registered [ 825.119404] Key type .llcrypt registered [ 825.391125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 825.419402] alg: No test for adler32 (adler32-zlib) [ 826.354270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 826.433566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 826.467482] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 827.072951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 827.091635] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 829.115841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 829.138860] LNet: 45156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 829.151803] LNetError: 45156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 829.158328] LNet: Removed LNI 10.1.2.3@tcp [ 829.170881] LNet: Removed LNI 192.168.203.36@tcp [ 829.199751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 829.276750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 829.306977] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 829.315706] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 829.865262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 829.908255] LNet: 45183:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 829.916774] LNetError: 45183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 829.929982] LNetError: 45183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 829.942681] LNet: Removed LNI 10.1.2.3@tcp [ 829.956951] LNet: Removed LNI 192.168.203.36@tcp [ 829.998783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 830.080790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 830.116142] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 830.131778] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 830.621872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 830.654334] LNet: Added LNI 192.168.203.36@tcp2 [8/256/0/180] [ 831.883557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 831.900866] LNet: 45345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 831.906236] LNetError: 45345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 831.911658] LNetError: 45345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 831.917395] LNet: Removed LNI 10.1.2.3@tcp [ 831.920352] LNet: Removed LNI 192.168.203.36@tcp [ 831.938635] LNet: Removed LNI 192.168.203.36@tcp2 [ 831.956725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 832.015303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 832.038426] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 832.043890] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 832.051099] LNet: Added LNI 192.168.203.36@tcp2 [8/256/0/180] [ 840.493830] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 17:24:07 (1788643447) [ 840.603881] LNet: 45935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 840.610627] LNetError: 45935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 840.625291] LNetError: 45935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 840.641757] LNet: Removed LNI 10.1.2.3@tcp [ 840.645923] LNet: Removed LNI 192.168.203.36@tcp [ 840.687463] LNet: Removed LNI 192.168.203.36@tcp2 [ 841.886100] Key type .llcrypt unregistered [ 841.892773] Key type ._llcrypt unregistered [ 848.752790] Key type ._llcrypt registered [ 848.758905] Key type .llcrypt registered [ 849.079657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 849.122442] alg: No test for adler32 (adler32-zlib) [ 850.100938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 850.172841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 850.209942] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 851.388222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 851.420066] LNet: 46335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 851.429107] LNetError: 46335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 851.444705] LNet: Removed LNI 192.168.203.36@tcp [ 851.482522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 851.563907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 851.627551] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 852.483733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 852.507834] LNet: 46365:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 852.514798] LNetError: 46365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 852.574509] LNet: Removed LNI 192.168.203.36@tcp [ 852.619058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 852.744573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 852.791911] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 853.170854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 853.215623] LNet: Removed LNI 192.168.203.36@tcp [ 853.239520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 853.344715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 853.403887] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 853.913900] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 853.961983] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 853.970958] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 853.978946] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 855.459472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 855.480994] LNet: 46466:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 855.485990] LNet: 46466:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 855.494049] LNetError: 46466:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 855.502690] LNetError: 46466:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 855.544601] LNet: Removed LNI 192.168.203.36@tcp [ 855.630314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 855.767151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 855.814465] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 865.414833] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 17:24:31 (1788643471) [ 865.484520] LNet: 47055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 865.491516] LNetError: 47055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 865.510030] LNet: Removed LNI 192.168.203.36@tcp [ 866.307425] Key type .llcrypt unregistered [ 866.311206] Key type ._llcrypt unregistered [ 873.435260] Key type ._llcrypt registered [ 873.437831] Key type .llcrypt registered [ 873.744462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 873.772754] alg: No test for adler32 (adler32-zlib) [ 874.820936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 874.882555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 874.904398] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 876.125215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 876.153550] LNet: 47460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 876.161882] LNetError: 47460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 876.187447] LNet: Removed LNI 192.168.203.36@tcp [ 876.212448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 876.348995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 876.397164] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 884.042947] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 17:24:50 (1788643490) [ 884.311591] LNet: 48052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 884.318490] LNetError: 48052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 884.338397] LNet: Removed LNI 192.168.203.36@tcp [ 885.046619] Key type .llcrypt unregistered [ 885.057627] Key type ._llcrypt unregistered [ 890.974492] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 891.298393] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 891.304925] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 891.317290] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 894.083145] Key type ._llcrypt registered [ 894.084526] Key type .llcrypt registered [ 894.291696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 894.320294] alg: No test for adler32 (adler32-zlib) [ 895.248382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.333787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 895.395919] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 896.598299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 896.630523] LNet: 48500:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 896.648517] LNetError: 48500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 896.671881] LNet: Removed LNI 192.168.203.36@tcp [ 896.721374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 896.794490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 896.829859] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 905.627683] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 17:25:11 (1788643511) [ 905.857336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 905.905395] LNet: 49096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 905.925483] LNetError: 49096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 905.977695] LNet: Removed LNI 192.168.203.36@tcp [ 906.057268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 906.191381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 913.599603] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 17:25:20 (1788643520) [ 913.768913] LNet: 49680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 914.510776] Key type .llcrypt unregistered [ 914.518106] Key type ._llcrypt unregistered [ 920.769078] Key type ._llcrypt registered [ 920.774646] Key type .llcrypt registered [ 921.021098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 921.033761] alg: No test for adler32 (adler32-zlib) [ 922.038995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 922.170499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 922.215860] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 923.005961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 923.058546] LNet: 50078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 923.065798] LNetError: 50078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 923.085954] LNet: Removed LNI 192.168.203.36@tcp [ 923.102536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 923.185608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 923.219862] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 930.695551] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 17:25:37 (1788643537) [ 930.826959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 930.856434] LNet: 50667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 930.869977] LNetError: 50667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 930.888945] LNet: Removed LNI 192.168.203.36@tcp [ 930.952229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 931.015255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 938.684633] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 17:25:45 (1788643545) [ 940.763917] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 942.927574] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 17:25:49 (1788643549) [ 944.596982] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 946.745213] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 17:25:53 (1788643553) [ 948.696206] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 950.538866] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 17:25:57 (1788643557) [ 952.316543] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 954.209822] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 17:26:00 (1788643560) [ 955.946742] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 957.851864] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 17:26:04 (1788643564) [ 957.959577] LNet: 52716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 958.721757] Key type .llcrypt unregistered [ 958.724213] Key type ._llcrypt unregistered [ 964.893408] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 965.044761] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 965.053611] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 965.068856] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 967.682142] Key type ._llcrypt registered [ 967.684409] Key type .llcrypt registered [ 967.922066] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 967.938744] alg: No test for adler32 (adler32-zlib) [ 968.986783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 969.115563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 969.146444] LNet: Added LNI 192.168.203.36@tcp [8/512/0/180] [ 969.151073] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 969.155194] LNet: 53041:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 969.270549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 969.293251] LNet: 53060:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 969.305724] LNetError: 53060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 969.312035] LNet: Removed LNI 10.1.2.3@tcp [ 969.315287] LNetError: 53060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 969.340581] LNet: Removed LNI 192.168.203.36@tcp [ 969.376567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 969.469896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 969.501362] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 969.601938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 969.632469] LNet: 53084:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 969.641286] LNetError: 53084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 969.665116] LNet: Removed LNI 192.168.203.36@tcp [ 969.696911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 969.795830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 969.839201] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 969.853979] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 969.856215] LNet: 53092:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 969.971979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 969.993589] LNet: 53110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 969.998991] LNetError: 53110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 970.004429] LNet: Removed LNI 10.1.2.3@tcp [ 970.006620] LNetError: 53110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 970.018058] LNet: Removed LNI 192.168.203.36@tcp [ 970.047792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 970.123875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 970.153429] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 970.159114] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 970.162284] LNet: 53118:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 970.353740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 970.381694] LNet: 53144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.390808] LNetError: 53144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 970.399180] LNet: Removed LNI 10.1.2.3@tcp [ 970.402769] LNetError: 53144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 970.416745] LNet: Removed LNI 192.168.203.36@tcp [ 970.450643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 970.515691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 970.556422] LNet: Added LNI 192.168.203.36@tcp [8/256/0/182] [ 970.565749] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 970.661252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 970.682606] LNet: 53172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.687408] LNetError: 53172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 970.691894] LNet: Removed LNI 10.1.2.3@tcp [ 970.693663] LNetError: 53172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 970.705770] LNet: Removed LNI 192.168.203.36@tcp [ 970.744961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 970.819946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 970.841369] LNet: Added LNI 192.168.203.36@tcp [8/256/0/182] [ 970.846056] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 970.847979] LNet: 53180:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 970.917763] LNet: Removed LNI 10.1.2.3@tcp [ 970.943443] LNet: Removed LNI 192.168.203.36@tcp [ 971.702933] Key type .llcrypt unregistered [ 971.705564] Key type ._llcrypt unregistered [ 983.521247] Key type ._llcrypt registered [ 983.525441] Key type .llcrypt registered [ 984.922174] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 17:26:31 (1788643591) [ 986.933638] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 989.252558] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 17:26:35 (1788643595) [ 989.734397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 989.752536] alg: No test for adler32 (adler32-zlib) [ 990.724399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 990.811529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 990.849347] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 991.940386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.36 [ 991.974126] LNet: 54443:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 991.982980] LNet: Removed LNI 192.168.203.36@tcp [ 997.641439] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 17:26:44 (1788643604) [ 997.848153] LNet: 55013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 997.855230] LNetError: 55013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 997.879110] LNet: Removed LNI 192.168.203.36@tcp [ 998.544602] Key type .llcrypt unregistered [ 998.547578] Key type ._llcrypt unregistered [ 1005.172290] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1005.398496] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1008.203854] Key type ._llcrypt registered [ 1008.210306] Key type .llcrypt registered [ 1008.415920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1008.434568] alg: No test for adler32 (adler32-zlib) [ 1009.774347] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1016.347245] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 17:27:03 (1788643623) [ 1016.525194] LNet: 55922:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1016.541398] LNetError: 55922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1016.559722] LNet: Removed LNI 10.1.2.3@tcp [ 1017.139552] Key type .llcrypt unregistered [ 1017.143851] Key type ._llcrypt unregistered [ 1022.896076] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1023.097433] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1023.113669] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1023.131739] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1025.920253] Key type ._llcrypt registered [ 1025.922031] Key type .llcrypt registered [ 1026.135634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1026.159987] alg: No test for adler32 (adler32-zlib) [ 1027.188150] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1034.238471] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 17:27:20 (1788643640) [ 1036.162512] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1038.866893] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 17:27:25 (1788643645) [ 1039.166892] LNet: 57105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1039.172637] LNetError: 57105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1039.199950] LNet: Removed LNI 10.1.2.3@tcp [ 1039.809667] Key type .llcrypt unregistered [ 1039.812816] Key type ._llcrypt unregistered [ 1046.288491] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1046.628622] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1046.632148] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1046.641474] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1048.335682] Key type ._llcrypt registered [ 1048.338905] Key type .llcrypt registered [ 1048.629356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1048.666563] alg: No test for adler32 (adler32-zlib) [ 1049.611795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1049.678596] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1059.869991] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 17:27:46 (1788643666) [ 1060.366731] LNet: 58325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1060.373565] LNetError: 58325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1060.398489] LNet: Removed LNI 10.1.2.3@tcp [ 1061.242941] Key type .llcrypt unregistered [ 1061.246339] Key type ._llcrypt unregistered [ 1067.617426] Key type ._llcrypt registered [ 1067.624178] Key type .llcrypt registered [ 1067.911741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1067.938083] alg: No test for adler32 (adler32-zlib) [ 1068.926728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1068.954088] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1075.610268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.136@tcp [ 1076.399207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp -m GET -e local_interrupt -r 1 [ 1076.447994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp -m GET -e local_interrupt -r 1 [ 1076.491592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.136@tcp [ 1076.520401] LNet: There was an unexpected network error while writing to 192.168.203.136: rc = -22 [ 1076.724489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1076.753625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1076.779586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1076.805809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1076.832916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1076.857283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1076.892685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1076.934681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1076.975103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1076.991138] LNet: 58761:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1077.016431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1077.050564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1077.103585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1077.132546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1077.151323] LNet: 1 local NIs in recovery (showing 1): 192.168.203.36@tcp [ 1077.155493] LNet: There was an unexpected network error while writing to 192.168.203.36: rc = -22 [ 1077.184177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1077.229876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1077.264860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1077.759384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp -m GET -e local_dropped -r 1 [ 1077.802203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp -m GET -e local_dropped -r 1 [ 1077.836951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.136@tcp [ 1078.040103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1078.071523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1078.096617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1078.129306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1078.152942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1078.197208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1078.225333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1078.255912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1078.283767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1078.305115] LNet: 58872:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1078.319280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1078.345311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1078.368563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1078.391957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1078.418907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1078.444640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1078.476934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1078.970698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp -m GET -e local_aborted -r 1 [ 1079.011503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp -m GET -e local_aborted -r 1 [ 1079.075511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.136@tcp [ 1079.107246] LNet: There was an unexpected network error while writing to 192.168.203.136: rc = -22 [ 1079.116642] LNet: Skipped 1 previous similar message [ 1079.379734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1079.420901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1079.469618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1079.498714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1079.552775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1079.590069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1079.634357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1079.667352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1079.701860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1079.720819] LNet: 58983:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1079.748924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1079.793851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1079.837524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1079.874623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1079.915263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1079.947613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1079.982481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1080.403101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp -m GET -e local_no_route -r 1 [ 1080.438968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp -m GET -e local_no_route -r 1 [ 1080.481626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.136@tcp [ 1080.685188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1080.709433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1080.739320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1080.768962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1080.806337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1080.848400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1080.882622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1080.932598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1080.989060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1081.046653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1081.080170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1081.134550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1081.176952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1081.215546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1081.239559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1081.276966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1081.687228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp -m GET -e local_timeout -r 1 [ 1081.726685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp -m GET -e local_timeout -r 1 [ 1081.771178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.136@tcp [ 1081.791601] LNet: There was an unexpected network error while writing to 192.168.203.136: rc = -22 [ 1081.797855] LNet: Skipped 2 previous similar messages [ 1081.985926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1082.019153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1082.044469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1082.081972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1082.105446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1082.132873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1082.164126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1082.185281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1082.215315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1082.226876] LNet: 59206:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1082.231830] LNet: 59206:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1082.249293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1082.279461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1082.315222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1082.343775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1082.385682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1082.427984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1082.470137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1082.958572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp -m GET -e local_error -r 1 [ 1083.006079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp -m GET -e local_error -r 1 [ 1083.052641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.136@tcp [ 1083.328822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1083.360110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1083.402385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1083.424032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1083.442813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1083.463938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1083.488315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1083.517909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1083.548529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1083.574122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1083.602619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1083.633621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1083.657361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1083.683978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1083.714065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1083.769311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1084.000661] LNet: 59358:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1084.006986] LNetError: 59358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1085.029691] LNet: Removed LNI 192.168.203.36@tcp [ 1085.380588] Key type .llcrypt unregistered [ 1085.384549] Key type ._llcrypt unregistered [ 1101.284264] Key type ._llcrypt registered [ 1101.288468] Key type .llcrypt registered [ 1102.528456] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 17:28:29 (1788643709) [ 1103.335283] Key type .llcrypt unregistered [ 1103.337447] Key type ._llcrypt unregistered [ 1108.996208] Key type ._llcrypt registered [ 1109.000070] Key type .llcrypt registered [ 1109.226477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1109.241475] alg: No test for adler32 (adler32-zlib) [ 1110.160315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1110.188235] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1118.043446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.136@tcp [ 1127.437894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1127.505465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1127.542333] LNet: Added LNI 192.168.203.36@tcp1 [8/256/0/180] [ 1129.228994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp -m GET -e local_interrupt -r 1 [ 1129.266647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp1 -m GET -e local_interrupt -r 1 [ 1129.294778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp -m GET -e local_interrupt -r 1 [ 1129.344812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp1 -m GET -e local_interrupt -r 1 [ 1129.376746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp1 -d 192.168.203.136@tcp -m GET -e local_interrupt -r 1 [ 1129.415756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp1 -d 192.168.203.136@tcp1 -m GET -e local_interrupt -r 1 [ 1129.450713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp1 -d 192.168.203.36@tcp -m GET -e local_interrupt -r 1 [ 1129.485809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp1 -d 192.168.203.36@tcp1 -m GET -e local_interrupt -r 1 [ 1129.523585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.136@tcp [ 1129.540059] LNet: There was an unexpected network error while writing to 192.168.203.136: rc = -22 [ 1129.546482] LNet: 1 local NIs in recovery (showing 1): 192.168.203.36@tcp [ 1129.749964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1129.780898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1129.819746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1129.859936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1129.902680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1129.933631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1129.966560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1129.990341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1130.015211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1130.031333] LNet: 60822:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1130.050394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1130.079908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1130.114255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1130.158051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1130.196340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1130.232683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1130.265490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1130.697729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp -m GET -e local_dropped -r 1 [ 1130.735587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp1 -m GET -e local_dropped -r 1 [ 1130.772338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp -m GET -e local_dropped -r 1 [ 1130.828665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp1 -m GET -e local_dropped -r 1 [ 1130.861582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp1 -d 192.168.203.136@tcp -m GET -e local_dropped -r 1 [ 1130.912548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp1 -d 192.168.203.136@tcp1 -m GET -e local_dropped -r 1 [ 1130.950377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp1 -d 192.168.203.36@tcp -m GET -e local_dropped -r 1 [ 1131.005851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp1 -d 192.168.203.36@tcp1 -m GET -e local_dropped -r 1 [ 1131.062842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.136@tcp [ 1131.084944] LNet: There was an unexpected network error while writing to 192.168.203.136: rc = -22 [ 1131.091970] LNet: Skipped 2 previous similar messages [ 1131.334343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1131.391753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1131.434339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1131.456981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1131.503912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1131.539656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1131.575325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1131.653828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1131.727682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1131.759933] LNet: 60945:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1131.782915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1131.824788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1131.860394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1131.903370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1131.942033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1131.976491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1132.015314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1132.611426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp -m GET -e local_aborted -r 1 [ 1132.658881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp1 -m GET -e local_aborted -r 1 [ 1132.693372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp -m GET -e local_aborted -r 1 [ 1132.724948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp1 -m GET -e local_aborted -r 1 [ 1132.750728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp1 -d 192.168.203.136@tcp -m GET -e local_aborted -r 1 [ 1132.776741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp1 -d 192.168.203.136@tcp1 -m GET -e local_aborted -r 1 [ 1132.810589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp1 -d 192.168.203.36@tcp -m GET -e local_aborted -r 1 [ 1132.845564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp1 -d 192.168.203.36@tcp1 -m GET -e local_aborted -r 1 [ 1132.873388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.136@tcp [ 1132.900448] LNet: There was an unexpected network error while writing to 192.168.203.136: rc = -22 [ 1132.904429] LNet: Skipped 2 previous similar messages [ 1133.091154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1133.120595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1133.171466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1133.207939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1133.275846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1133.311738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1133.352471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1133.398570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1133.433446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1133.485600] LNet: 61068:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1133.509833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1133.544883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1133.589515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1133.652345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1133.696844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1133.752985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1133.786421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1134.207784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp -m GET -e local_no_route -r 1 [ 1134.259680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp1 -m GET -e local_no_route -r 1 [ 1134.292789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp -m GET -e local_no_route -r 1 [ 1134.324491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp1 -m GET -e local_no_route -r 1 [ 1134.350846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp1 -d 192.168.203.136@tcp -m GET -e local_no_route -r 1 [ 1134.380805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp1 -d 192.168.203.136@tcp1 -m GET -e local_no_route -r 1 [ 1134.408792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp1 -d 192.168.203.36@tcp -m GET -e local_no_route -r 1 [ 1134.461989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp1 -d 192.168.203.36@tcp1 -m GET -e local_no_route -r 1 [ 1134.506112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.136@tcp [ 1134.717641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1134.753047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1134.795244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1134.827159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1134.878762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1134.920644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1134.964489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1135.012279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1135.061318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1135.093858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1135.127094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1135.184862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1135.246197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1135.287634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1135.335816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1135.390216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1135.872971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp -m GET -e local_timeout -r 1 [ 1135.908690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp1 -m GET -e local_timeout -r 1 [ 1135.940289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp -m GET -e local_timeout -r 1 [ 1135.973663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp1 -m GET -e local_timeout -r 1 [ 1136.007539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp1 -d 192.168.203.136@tcp -m GET -e local_timeout -r 1 [ 1136.041873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp1 -d 192.168.203.136@tcp1 -m GET -e local_timeout -r 1 [ 1136.081543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp1 -d 192.168.203.36@tcp -m GET -e local_timeout -r 1 [ 1136.121550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp1 -d 192.168.203.36@tcp1 -m GET -e local_timeout -r 1 [ 1136.159920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.136@tcp [ 1136.180522] LNet: There was an unexpected network error while writing to 192.168.203.136: rc = -22 [ 1136.186910] LNet: Skipped 5 previous similar messages [ 1136.360774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1136.394879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1136.446835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1136.487455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1136.521627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1136.555675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1136.616080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1136.688226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1136.748483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1136.781835] LNet: 61315:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1136.799305] LNet: 61315:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1136.822417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1136.873969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1136.920235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1136.955115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1136.977366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1137.000233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1137.092799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1137.770244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp -m GET -e local_error -r 1 [ 1137.821403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp1 -m GET -e local_error -r 1 [ 1137.850406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp -m GET -e local_error -r 1 [ 1137.880188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp1 -m GET -e local_error -r 1 [ 1137.913643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp1 -d 192.168.203.136@tcp -m GET -e local_error -r 1 [ 1137.963896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp1 -d 192.168.203.136@tcp1 -m GET -e local_error -r 1 [ 1137.999658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp1 -d 192.168.203.36@tcp -m GET -e local_error -r 1 [ 1138.043642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp1 -d 192.168.203.36@tcp1 -m GET -e local_error -r 1 [ 1138.093541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.136@tcp [ 1138.408357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1138.439429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1138.463447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1138.486749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1138.512262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1138.534619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1138.568244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1138.611872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1138.637114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1138.666074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1138.695137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1138.716620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1138.772508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1138.819121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1138.883953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1138.935621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1145.300482] LNet: 61514:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1145.305741] LNetError: 61514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1145.311717] LNet: Removed LNI 192.168.203.36@tcp [ 1145.325675] LNet: Removed LNI 192.168.203.36@tcp1 [ 1145.885322] Key type .llcrypt unregistered [ 1145.887467] Key type ._llcrypt unregistered [ 1164.732641] Key type ._llcrypt registered [ 1164.734479] Key type .llcrypt registered [ 1166.250396] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 17:29:32 (1788643772) [ 1167.663673] Key type .llcrypt unregistered [ 1167.670044] Key type ._llcrypt unregistered [ 1176.037659] Key type ._llcrypt registered [ 1176.040531] Key type .llcrypt registered [ 1176.202180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1176.221751] alg: No test for adler32 (adler32-zlib) [ 1177.282514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1177.382906] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1186.173512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.136@tcp [ 1186.909186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp -m GET -e remote_dropped -r 1 [ 1186.940276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp -m GET -e remote_dropped -r 1 [ 1186.972761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.136@tcp [ 1186.992295] LNet: There was an unexpected network error while writing to 192.168.203.136: rc = -22 [ 1187.244276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1187.280669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1187.308353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1187.345719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1187.374694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1187.404378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1187.451956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1187.507787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1187.541374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1187.564312] LNet: 62777:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1187.598225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1187.640190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1187.668506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1187.709613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1187.742564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1187.770161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1187.811764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1188.296924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp -m GET -e remote_error -r 1 [ 1188.335110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp -m GET -e remote_error -r 1 [ 1188.370087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.136@tcp [ 1188.392757] LNet: There was an unexpected network error while writing to 192.168.203.136: rc = -22 [ 1188.631260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1188.679623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1188.741205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1188.813661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1188.847159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1188.887035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1188.918818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1188.951399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1188.983723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1188.999661] LNet: 62888:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1189.033331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1189.090344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1189.159380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1189.214776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1189.263492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1189.311795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1189.363054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1189.913780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp -m GET -e remote_timeout -r 1 [ 1189.954745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp -m GET -e remote_timeout -r 1 [ 1189.994861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.136@tcp [ 1190.013882] LNet: There was an unexpected network error while writing to 192.168.203.136: rc = -22 [ 1190.208909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1190.236255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1190.266819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1190.298413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1190.335580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1190.375583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1190.414924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1190.444409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1190.479101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1190.502876] LNet: 62999:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1190.533089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1190.597120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1190.636529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1190.669939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1190.711481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1190.769245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1190.825698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1191.074062] LNet: 63040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1191.079520] LNetError: 63040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1191.096307] LNet: Removed LNI 192.168.203.36@tcp [ 1191.435629] Key type .llcrypt unregistered [ 1191.440720] Key type ._llcrypt unregistered [ 1207.564666] Key type ._llcrypt registered [ 1207.566981] Key type .llcrypt registered [ 1208.848035] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 17:30:15 (1788643815) [ 1209.657397] Key type .llcrypt unregistered [ 1209.659751] Key type ._llcrypt unregistered [ 1216.042105] Key type ._llcrypt registered [ 1216.048536] Key type .llcrypt registered [ 1216.326565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1216.359612] alg: No test for adler32 (adler32-zlib) [ 1217.261437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1217.282372] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1224.785529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.136@tcp [ 1234.214592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1234.282870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1234.339462] LNet: Added LNI 192.168.203.36@tcp1 [8/256/0/180] [ 1235.913234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp -m GET -e remote_dropped -r 1 [ 1235.977321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp1 -m GET -e remote_dropped -r 1 [ 1236.033143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp -m GET -e remote_dropped -r 1 [ 1236.087108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp1 -m GET -e remote_dropped -r 1 [ 1236.120392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp1 -d 192.168.203.136@tcp -m GET -e remote_dropped -r 1 [ 1236.154807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp1 -d 192.168.203.136@tcp1 -m GET -e remote_dropped -r 1 [ 1236.201690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp1 -d 192.168.203.36@tcp -m GET -e remote_dropped -r 1 [ 1236.245113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp1 -d 192.168.203.36@tcp1 -m GET -e remote_dropped -r 1 [ 1236.300406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.136@tcp [ 1236.320728] LNet: There was an unexpected network error while writing to 192.168.203.136: rc = -22 [ 1236.328314] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.136@tcp [ 1236.584872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1236.626604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1236.681457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1236.722262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1236.800664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1236.846532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1236.894734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1236.984269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1237.029382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1237.060048] LNet: 64504:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1237.102648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1237.166691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1237.233699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1237.303588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1237.342951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1237.344535] LNet: There was an unexpected network error while writing to 192.168.203.136: rc = -22 [ 1237.362208] LNet: Skipped 3 previous similar messages [ 1237.403524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1237.453585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1237.532345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1238.359593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp -m GET -e remote_error -r 1 [ 1238.416845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp1 -m GET -e remote_error -r 1 [ 1238.469730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp -m GET -e remote_error -r 1 [ 1238.506376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp1 -m GET -e remote_error -r 1 [ 1238.543787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp1 -d 192.168.203.136@tcp -m GET -e remote_error -r 1 [ 1238.598444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp1 -d 192.168.203.136@tcp1 -m GET -e remote_error -r 1 [ 1238.648513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp1 -d 192.168.203.36@tcp -m GET -e remote_error -r 1 [ 1238.709528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp1 -d 192.168.203.36@tcp1 -m GET -e remote_error -r 1 [ 1238.747353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.136@tcp [ 1238.766905] LNet: There was an unexpected network error while writing to 192.168.203.136: rc = -22 [ 1239.173860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1239.215924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1239.273410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1239.303879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1239.371108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1239.447365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1239.513585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1239.616083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1239.717053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1239.766616] LNet: 64629:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1239.796453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1239.866097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1239.983390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1240.050335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1240.076475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1240.122636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1240.178453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1240.255123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1241.017767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp -m GET -e remote_timeout -r 1 [ 1241.043827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp1 -m GET -e remote_timeout -r 1 [ 1241.069542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp -m GET -e remote_timeout -r 1 [ 1241.092778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp1 -m GET -e remote_timeout -r 1 [ 1241.120933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp1 -d 192.168.203.136@tcp -m GET -e remote_timeout -r 1 [ 1241.147270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp1 -d 192.168.203.136@tcp1 -m GET -e remote_timeout -r 1 [ 1241.177748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp1 -d 192.168.203.36@tcp -m GET -e remote_timeout -r 1 [ 1241.204566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp1 -d 192.168.203.36@tcp1 -m GET -e remote_timeout -r 1 [ 1241.235589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.136@tcp [ 1241.253190] LNet: There was an unexpected network error while writing to 192.168.203.136: rc = -22 [ 1241.434292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1241.465448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1241.492916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1241.519494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1241.549602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1241.613206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1241.680568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1241.750298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1241.835411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1241.882274] LNet: 64755:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1241.941702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1241.978851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1242.047600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1242.115370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1242.182349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1242.219326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1242.286909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1242.342979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1247.023577] LNet: 64833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1247.035294] LNetError: 64833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1247.044984] LNet: Removed LNI 192.168.203.36@tcp [ 1247.066691] LNet: Removed LNI 192.168.203.36@tcp1 [ 1247.539637] Key type .llcrypt unregistered [ 1247.543422] Key type ._llcrypt unregistered [ 1267.331205] Key type ._llcrypt registered [ 1267.333183] Key type .llcrypt registered [ 1269.208748] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 17:31:15 (1788643875) [ 1270.991559] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1272.770439] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 17:31:19 (1788643879) [ 1273.792299] Key type .llcrypt unregistered [ 1273.793870] Key type ._llcrypt unregistered [ 1280.486918] Key type ._llcrypt registered [ 1280.489943] Key type .llcrypt registered [ 1280.727063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1280.755679] alg: No test for adler32 (adler32-zlib) [ 1281.726368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1281.767473] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1290.540106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.136@tcp [ 1291.596892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp -m GET -e network_timeout -r 1 [ 1291.670723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp -m GET -e network_timeout -r 1 [ 1291.726893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.136@tcp [ 1291.750869] LNet: There was an unexpected network error while writing to 192.168.203.136: rc = -22 [ 1292.014401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1292.040829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1292.067593] LNet: 1 local NIs in recovery (showing 1): 192.168.203.36@tcp [ 1292.072112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1292.115424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1292.160416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1292.199527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1292.238979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1292.285578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1292.326222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1292.339245] LNet: 66373:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1292.357855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1292.399963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1292.488687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1292.543722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1292.597280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1292.641461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1292.997865] LNet: 66413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1293.003547] LNetError: 66413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1294.054497] LNet: Removed LNI 192.168.203.36@tcp [ 1294.621814] Key type .llcrypt unregistered [ 1294.624650] Key type ._llcrypt unregistered [ 1314.768807] Key type ._llcrypt registered [ 1314.773425] Key type .llcrypt registered [ 1315.075582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1315.117960] alg: No test for adler32 (adler32-zlib) [ 1316.061489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1316.103191] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1324.231868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.136@tcp [ 1333.531276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1333.571443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1333.590436] LNet: Added LNI 192.168.203.36@tcp1 [8/256/0/180] [ 1335.373126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp -m GET -e network_timeout -r 1 [ 1335.412874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp1 -m GET -e network_timeout -r 1 [ 1335.450890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp -m GET -e network_timeout -r 1 [ 1335.485248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp1 -m GET -e network_timeout -r 1 [ 1335.521332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp1 -d 192.168.203.136@tcp -m GET -e network_timeout -r 1 [ 1335.565889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp1 -d 192.168.203.136@tcp1 -m GET -e network_timeout -r 1 [ 1335.625821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp1 -d 192.168.203.36@tcp -m GET -e network_timeout -r 1 [ 1335.686706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp1 -d 192.168.203.36@tcp1 -m GET -e network_timeout -r 1 [ 1335.742101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.136@tcp [ 1335.770955] LNet: There was an unexpected network error while writing to 192.168.203.136: rc = -22 [ 1336.028358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1336.055243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1336.088182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1336.113615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1336.140664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1336.167246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1336.195872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1336.228537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1336.263991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1336.282444] LNet: 67289:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1336.312725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1336.339594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1336.367931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1336.402392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1336.434608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1336.472707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1341.444772] LNet: There was an unexpected network error while writing to 192.168.203.136: rc = -22 [ 1341.730853] LNet: 1 local NIs in recovery (showing 1): 192.168.203.36@tcp [ 1341.752722] LNet: 67363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1341.762126] LNetError: 67363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1341.772799] LNet: Removed LNI 192.168.203.36@tcp [ 1341.786883] LNet: Removed LNI 192.168.203.36@tcp1 [ 1342.152840] Key type .llcrypt unregistered [ 1342.154678] Key type ._llcrypt unregistered [ 1360.251960] Key type ._llcrypt registered [ 1360.255765] Key type .llcrypt registered [ 1361.514389] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 17:32:48 (1788643968) [ 1362.042579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1362.062201] alg: No test for adler32 (adler32-zlib) [ 1362.967265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1363.016609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1363.053096] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1363.928947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1363.962640] LNet: Added LNI 192.168.203.36@tcp1 [8/256/0/180] [ 1364.953947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.36@tcp [ 1365.046438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1365.100186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1365.172818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1365.224570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1365.271125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1365.354267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.36@tcp [ 1365.382701] LNetError: 68460:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.36@tcp to recovery [ 1366.048717] LNet: 1 local NIs in recovery (showing 1): 192.168.203.36@tcp [ 1366.048782] LNet: There was an unexpected network error while writing to 192.168.203.36: rc = -22 [ 1368.159938] LNet: There was an unexpected network error while writing to 192.168.203.36: rc = -22 [ 1369.048895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1372.257360] LNet: There was an unexpected network error while writing to 192.168.203.36: rc = -22 [ 1372.629075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1372.661735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1372.707359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1372.740903] LNet: 68568:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1372.745854] LNetError: 68568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1372.760137] LNet: Removed LNI 192.168.203.36@tcp [ 1372.773024] LNet: Removed LNI 192.168.203.36@tcp1 [ 1372.813543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1372.867463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1372.895408] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1373.680426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1373.694464] LNet: Added LNI 192.168.203.36@tcp1 [8/256/0/180] [ 1374.427213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.36@tcp [ 1374.466832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1374.522992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1374.586457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1374.638719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1374.676906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1374.707319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1374.734623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.36@tcp [ 1374.752031] LNetError: 68840:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.36@tcp to recovery [ 1374.880516] LNet: There was an unexpected network error while writing to 192.168.203.36: rc = -22 [ 1377.191948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1381.023713] LNet: There was an unexpected network error while writing to 192.168.203.36: rc = -22 [ 1381.038460] LNet: Skipped 1 previous similar message [ 1385.318838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1385.357481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1385.387979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1392.449384] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 17:33:19 (1788643999) [ 1392.748030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1392.775207] LNet: 69546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1392.786963] LNetError: 69546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1392.798984] LNetError: 69546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1392.815902] LNet: Removed LNI 192.168.203.36@tcp [ 1392.837799] LNet: Removed LNI 192.168.203.36@tcp1 [ 1392.899864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1392.998632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1393.044036] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1393.840933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1393.863704] LNet: Added LNI 192.168.203.36@tcp1 [8/256/0/180] [ 1394.675615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.36@tcp [ 1394.733636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1394.770391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1394.827183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1394.884659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1394.919626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1394.996219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.36@tcp --health 0 [ 1395.207302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1395.999210] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.36@tcp [ 1397.024493] LNet: There was an unexpected network error while writing to 192.168.203.36: rc = -22 [ 1397.040116] LNet: Skipped 1 previous similar message [ 1412.011895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1412.269883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1412.332306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1412.422760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.36@tcp --health 500 [ 1412.628202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1419.398957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1419.518985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1419.539661] LNet: 70047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1419.546978] LNetError: 70047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1419.550793] LNetError: 70047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1419.561644] LNet: Removed LNI 192.168.203.36@tcp [ 1419.573686] LNet: Removed LNI 192.168.203.36@tcp1 [ 1419.610303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1419.712277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1419.776949] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1420.688845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1420.744982] LNet: Added LNI 192.168.203.36@tcp1 [8/256/0/180] [ 1422.096675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.36@tcp [ 1422.151367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1422.253385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1422.305824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1422.375870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1422.453672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1422.515300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1422.585641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.36@tcp --health 0 [ 1423.775425] LNet: There was an unexpected network error while writing to 192.168.203.36: rc = -22 [ 1423.786793] LNet: Skipped 3 previous similar messages [ 1424.048348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1434.674221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1434.714229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1434.765996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1442.446398] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 17:34:09 (1788644049) [ 1443.228526] LNet: 71093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1443.238275] LNetError: 71093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1443.248717] LNetError: 71093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1443.256848] LNet: Removed LNI 192.168.203.36@tcp [ 1443.268551] LNet: Removed LNI 192.168.203.36@tcp1 [ 1443.910407] Key type .llcrypt unregistered [ 1443.912256] Key type ._llcrypt unregistered [ 1451.562861] Key type ._llcrypt registered [ 1451.564309] Key type .llcrypt registered [ 1451.914387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1451.931659] alg: No test for adler32 (adler32-zlib) [ 1452.977609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1453.036486] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1464.115499] Lustre: DEBUG MARKER: Initial discovery [ 1464.247668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.136@tcp [ 1466.600108] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1466.714929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1466.750270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.136@tcp [ 1466.768954] LNet: There was an unexpected network error while writing to 192.168.203.136: rc = -22 [ 1466.795600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1466.836797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.136@tcp -r 1 -m GET -l 3 [ 1466.862412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.136@tcp -r 1 -m GET -e local_error [ 1466.895431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.136@tcp -r 1 -m PUT -l 6 [ 1467.359422] LNet: 1 local NIs in recovery (showing 1): 192.168.203.36@tcp [ 1468.158262] Lustre: DEBUG MARKER: Force oleg336-client.virtnet to discover 192.168.203.136@tcp (in background) [ 1468.369253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.136@tcp [ 1469.669637] Lustre: DEBUG MARKER: Force oleg336-server to discover 192.168.203.36@tcp [ 1471.456899] LNet: There was an unexpected network error while writing to 192.168.203.136: rc = -22 [ 1472.437338] Lustre: DEBUG MARKER: Wait for 71854 [ 1474.437204] Lustre: DEBUG MARKER: Finished wait on 71854 [ 1474.510812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1474.550751] LNet: There was an unexpected network error while writing to 192.168.203.136: rc = -22 [ 1474.552170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1474.923973] LNet: 72302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1474.929781] LNetError: 72302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1474.953148] LNet: Removed LNI 192.168.203.36@tcp [ 1475.531955] Key type .llcrypt unregistered [ 1475.537929] Key type ._llcrypt unregistered [ 1495.487729] Key type ._llcrypt registered [ 1495.495652] Key type .llcrypt registered [ 1497.167950] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 17:35:03 (1788644103) [ 1497.795279] Key type .llcrypt unregistered [ 1497.797076] Key type ._llcrypt unregistered [ 1504.706813] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1504.827600] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1504.834725] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1504.839263] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1507.686586] Key type ._llcrypt registered [ 1507.689615] Key type .llcrypt registered [ 1507.910690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1507.929416] alg: No test for adler32 (adler32-zlib) [ 1508.913547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1509.065385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1509.116714] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1509.943229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1509.988486] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1519.781673] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 17:35:26 (1788644126) [ 1519.977560] LNet: 74374:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1519.999747] LNetError: 74374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1520.014272] LNet: Removed LNI 10.1.2.3@tcp [ 1520.031375] LNet: Removed LNI 192.168.203.36@tcp [ 1520.982402] Key type .llcrypt unregistered [ 1520.985105] Key type ._llcrypt unregistered [ 1528.736563] Key type ._llcrypt registered [ 1528.742737] Key type .llcrypt registered [ 1529.171046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1529.216922] alg: No test for adler32 (adler32-zlib) [ 1530.352109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1530.452391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1530.534576] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1531.726647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1531.766482] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1533.067133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.36@tcp [ 1534.177274] LNetError: Unexpected error -22 connecting to 192.168.203.36@tcp at host 192.168.203.36:988 [ 1540.552817] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 17:35:47 (1788644147) [ 1540.701047] LNet: 75501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1540.711205] LNetError: 75501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1540.716031] LNet: Removed LNI 10.1.2.3@tcp [ 1540.738148] LNet: Removed LNI 192.168.203.36@tcp [ 1541.614879] Key type .llcrypt unregistered [ 1541.619106] Key type ._llcrypt unregistered [ 1548.667793] Key type ._llcrypt registered [ 1548.670391] Key type .llcrypt registered [ 1548.834175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1548.850421] alg: No test for adler32 (adler32-zlib) [ 1549.784857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1549.855797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1549.873362] LNet: Added LNI 192.168.203.36@tcp1 [8/256/0/180] [ 1550.689233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1550.711033] LNet: Added LNI 192.168.203.36@tcp2 [8/256/0/180] [ 1551.490956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.36@tcp1 --nid 192.168.203.36@tcp2 [ 1555.368261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.36@tcp1 192.168.203.36@tcp2 [ 1555.388767] LNetError: 76337:0:(lib-move.c:2112:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.36@tcp1 to 192.168.203.36@tcp2 - Route table may be misconfigured [ 1555.406720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.36@tcp2 192.168.203.36@tcp1 [ 1562.754885] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 17:36:09 (1788644169) [ 1562.876753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1562.916730] LNet: 76902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1562.924419] LNetError: 76902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1562.930980] LNet: Removed LNI 192.168.203.36@tcp1 [ 1562.952347] LNet: Removed LNI 192.168.203.36@tcp2 [ 1563.011754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1563.092779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1563.118357] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1563.865938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1563.885527] LNet: Added LNI 192.168.203.36@tcp1 [8/256/0/180] [ 1565.025827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.36@tcp [ 1565.059801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.36@tcp 192.168.203.36@tcp [ 1565.107985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.36@tcp1 192.168.203.36@tcp1 [ 1565.151322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.36@tcp -d 192.168.203.36@tcp -e network_timeout [ 1565.204196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.36@tcp -d 192.168.203.36@tcp1 -e network_timeout [ 1565.240685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.36@tcp1 -d 192.168.203.36@tcp -e network_timeout [ 1565.303885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.36@tcp1 -d 192.168.203.36@tcp1 -e network_timeout [ 1565.389623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 1565.419852] LNet: There was an unexpected network error while writing to 192.168.203.36: rc = -22 [ 1565.453221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1565.596824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1565.732066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1566.111327] LNet: 1 local NIs in recovery (showing 1): 192.168.203.36@tcp [ 1573.999374] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 17:36:20 (1788644180) [ 1574.297347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1574.354697] LNet: 77773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1574.366628] LNetError: 77773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1574.375245] LNetError: 77773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1574.391539] LNet: Removed LNI 192.168.203.36@tcp [ 1575.474654] LNet: Removed LNI 192.168.203.36@tcp1 [ 1575.529575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1575.696087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1576.225254] LNet: 77810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1576.934878] Key type .llcrypt unregistered [ 1576.936809] Key type ._llcrypt unregistered [ 1591.808326] Key type ._llcrypt registered [ 1591.812672] Key type .llcrypt registered [ 1593.890613] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 17:36:39 (1788644199) [ 1594.460608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1594.489398] alg: No test for adler32 (adler32-zlib) [ 1595.455524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1597.091141] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1599.084136] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 17:36:45 (1788644205) [ 1599.328632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1599.344558] LNet: 78906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1599.360518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1599.405457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1599.422312] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1600.076627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1600.096686] LNet: Added LNI 192.168.203.36@tcp1 [8/256/0/180] [ 1601.056404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 1601.111383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp1 [ 1601.169568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.36@tcp1 [ 1608.212862] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 17:36:54 (1788644214) [ 1610.019745] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg336-server [ 1612.566202] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 17:36:58 (1788644218) [ 1614.616577] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg336-server [ 1616.672862] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 17:37:03 (1788644223) [ 1618.564530] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg336-server [ 1620.582671] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 17:37:07 (1788644227) [ 1622.881623] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg336-server [ 1624.961805] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 17:37:11 (1788644231) [ 1626.794642] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg336-server [ 1628.639631] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 17:37:15 (1788644235) [ 1630.917756] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg336-server [ 1632.886304] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 17:37:19 (1788644239) [ 1635.577535] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg336-server [ 1637.568488] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 17:37:24 (1788644244) [ 1639.859881] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg336-server [ 1642.254949] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 17:37:28 (1788644248) [ 1644.917037] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg336-server [ 1647.288712] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 17:37:33 (1788644253) [ 1647.347922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1647.361848] LNet: 82776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1647.365375] LNetError: 82776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1647.368979] LNet: Removed LNI 192.168.203.36@tcp [ 1648.423915] LNet: Removed LNI 192.168.203.36@tcp1 [ 1648.473490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1648.578848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1648.625973] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1649.461899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1649.567666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 1649.816376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1649.870232] LNet: 82931:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1649.884624] LNetError: 82931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1649.906569] LNetError: 82931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1649.931205] LNet: Removed LNI 192.168.203.36@tcp [ 1649.980441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1650.076361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1650.138520] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1651.266278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 1651.688617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1651.792954] LNetError: 83076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1651.837929] LNet: Removed LNI 192.168.203.36@tcp [ 1651.898334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1652.014152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1652.125869] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1653.990110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1654.196550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 1654.613507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1654.630725] LNet: 83224:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1654.635689] LNet: 83224:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1654.642622] LNetError: 83224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1655.659866] LNet: Removed LNI 192.168.203.36@tcp [ 1655.763768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1655.858735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1655.918617] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1657.125453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 1657.391514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1658.480438] LNet: Removed LNI 192.168.203.36@tcp [ 1658.527965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1658.601393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1658.657054] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1659.540177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1659.647803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 1659.861961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1659.910267] LNet: 83516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1659.915639] LNet: 83516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1659.931344] LNetError: 83516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1659.938119] LNetError: 83516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1659.963906] LNet: Removed LNI 192.168.203.36@tcp [ 1659.984814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1660.066931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1660.101106] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1661.040346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 1661.374645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1662.444501] LNet: Removed LNI 192.168.203.36@tcp [ 1662.467644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1662.606848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1662.644576] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1663.556834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1663.647583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 1663.874940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1665.010986] LNet: Removed LNI 192.168.203.36@tcp [ 1665.042221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1665.185544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1665.235619] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1666.404372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 1668.128064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1668.158809] LNet: 83963:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1668.167763] LNet: 83963:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1668.187065] LNetError: 83963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1668.198192] LNetError: 83963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1669.227689] LNet: Removed LNI 192.168.203.36@tcp [ 1669.319329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1669.450634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1669.476709] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1670.335893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1670.458115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 1670.755892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1671.846948] LNet: Removed LNI 192.168.203.36@tcp [ 1671.880471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1671.968571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1671.999953] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1672.935987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 1674.505355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1675.559277] LNet: Removed LNI 192.168.203.36@tcp [ 1675.597914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1675.701426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1675.733378] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1676.714490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1676.857167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 1677.201788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1678.278034] LNet: Removed LNI 192.168.203.36@tcp [ 1678.316716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1678.411053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1678.453849] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1679.269381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 1679.683456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1680.743263] LNet: Removed LNI 192.168.203.36@tcp [ 1680.795622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1680.917298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1680.953411] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1681.794619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1681.869761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1681.912575] LNet: Removed LNI 192.168.203.36@tcp [ 1681.926390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1682.007190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1682.030086] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1682.784679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1682.890990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1682.961084] LNet: Removed LNI 192.168.203.36@tcp [ 1682.987018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1683.041364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1683.132277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1683.165780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1683.258373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1683.402543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1683.464175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1683.542551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1683.622572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1683.660836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1683.740738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1683.780805] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1684.550287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1684.581365] LNet: 84998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1684.587084] LNet: 84998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1684.593938] LNetError: 84998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1684.600989] LNetError: 84998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1684.620065] LNet: Removed LNI 192.168.203.36@tcp [ 1684.641689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1684.720301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1692.471522] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 17:38:19 (1788644299) [ 1692.566113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1692.617067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1692.725416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1692.755369] LNet: Added LNI 192.168.203.36@tcp231 [8/256/0/180] [ 1694.169577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1694.232107] LNet: Removed LNI 192.168.203.36@tcp231 [ 1694.259863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1694.343442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1694.386521] LNet: Added LNI 192.168.203.36@tcp231 [8/256/0/180] [ 1694.397761] LNet: 85725:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1694.813849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1694.841626] LNet: Removed LNI 192.168.203.36@tcp231 [ 1694.858211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1694.882631] LNet: Added LNI 192.168.203.36@tcp231 [8/256/0/180] [ 1696.232700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1696.266450] LNet: Removed LNI 192.168.203.36@tcp231 [ 1696.302853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1696.382979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1696.416661] LNet: Added LNI 192.168.203.36@tcp231 [8/256/0/0] [ 1696.422775] LNet: 85877:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1696.881310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1696.923670] LNet: Removed LNI 192.168.203.36@tcp231 [ 1696.954227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1696.979834] LNet: Added LNI 192.168.203.36@tcp231 [8/256/0/0] [ 1698.353545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1698.408449] LNet: Removed LNI 192.168.203.36@tcp231 [ 1698.433941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1698.497859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1698.537227] LNet: Added LNI 192.168.203.36@tcp231 [8/256/0/60] [ 1698.554288] LNet: 86029:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1699.092481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1699.128639] LNet: Removed LNI 192.168.203.36@tcp231 [ 1699.150139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1699.179978] LNet: Added LNI 192.168.203.36@tcp231 [8/256/0/60] [ 1700.350716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1700.388239] LNet: Removed LNI 192.168.203.36@tcp231 [ 1700.406816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1700.482513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1700.521350] LNet: Added LNI 192.168.203.36@tcp231 [8/256/0/180] [ 1700.979253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1701.015650] LNet: Removed LNI 192.168.203.36@tcp231 [ 1701.058593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1701.109344] LNet: Added LNI 192.168.203.36@tcp231 [8/256/0/180] [ 1710.964237] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 17:38:37 (1788644317) [ 1711.569582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1711.645581] LNet: Removed LNI 192.168.203.36@tcp231 [ 1711.692821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1711.872703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1711.913994] LNet: Added LNI 192.168.203.36@tcp232 [8/256/0/180] [ 1712.821993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1712.903439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1712.946332] LNet: Removed LNI 192.168.203.36@tcp232 [ 1712.983642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1713.003221] LNet: Added LNI 192.168.203.36@tcp232 [8/256/0/180] [ 1714.082105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1714.114753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1714.143258] LNet: Removed LNI 192.168.203.36@tcp232 [ 1714.160205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1714.187110] LNet: Added LNI 192.168.203.36@tcp232 [8/256/0/180] [ 1715.028651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1715.078331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1715.108755] LNet: Removed LNI 192.168.203.36@tcp232 [ 1715.123501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1715.146544] LNet: Added LNI 192.168.203.36@tcp232 [8/256/0/180] [ 1715.903122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1715.971537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1716.040480] LNet: Removed LNI 192.168.203.36@tcp232 [ 1716.071351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1726.574735] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 17:38:52 (1788644332) [ 1727.452867] LNet: 88020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1727.461931] LNet: 88020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 1728.721666] Key type .llcrypt unregistered [ 1728.726389] Key type ._llcrypt unregistered [ 1736.975865] Key type ._llcrypt registered [ 1736.977477] Key type .llcrypt registered [ 1737.209568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1737.231026] alg: No test for adler32 (adler32-zlib) [ 1738.405857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1738.482932] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1747.947896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.136@tcp [ 1761.869660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1761.997811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1762.060082] LNet: Added LNI 192.168.203.36@tcp1 [8/256/0/180] [ 1765.382664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp -m GET -e local_interrupt -r 2 [ 1765.447940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp1 -m GET -e local_interrupt -r 2 [ 1765.517931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp -m GET -e local_interrupt -r 2 [ 1765.578996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp1 -m GET -e local_interrupt -r 2 [ 1765.630390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp1 -d 192.168.203.136@tcp -m GET -e local_interrupt -r 2 [ 1765.688312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp1 -d 192.168.203.136@tcp1 -m GET -e local_interrupt -r 2 [ 1765.754416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp1 -d 192.168.203.36@tcp -m GET -e local_interrupt -r 2 [ 1765.851777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp1 -d 192.168.203.36@tcp1 -m GET -e local_interrupt -r 2 [ 1765.881730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.36@tcp 192.168.203.136@tcp [ 1765.901352] LNet: There was an unexpected network error while writing to 192.168.203.136: rc = -22 [ 1765.905330] LNet: 1 local NIs in recovery (showing 1): 192.168.203.36@tcp [ 1766.144781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1766.204529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1766.245786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1766.299774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1766.404173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1766.462928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1766.533782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1766.577923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1766.663801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1766.686806] LNet: 88682:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1766.721852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1766.775793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1766.840763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1766.933435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1766.969763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1767.005938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1767.955261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp -m GET -e local_dropped -r 2 [ 1768.037546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp1 -m GET -e local_dropped -r 2 [ 1768.090993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp -m GET -e local_dropped -r 2 [ 1768.136852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp1 -m GET -e local_dropped -r 2 [ 1768.190847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp1 -d 192.168.203.136@tcp -m GET -e local_dropped -r 2 [ 1768.250630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp1 -d 192.168.203.136@tcp1 -m GET -e local_dropped -r 2 [ 1768.319900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp1 -d 192.168.203.36@tcp -m GET -e local_dropped -r 2 [ 1768.419810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp1 -d 192.168.203.36@tcp1 -m GET -e local_dropped -r 2 [ 1768.494919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.36@tcp 192.168.203.136@tcp [ 1768.516507] LNet: There was an unexpected network error while writing to 192.168.203.136: rc = -22 [ 1769.005662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1769.055434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1769.140165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1769.220701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1769.250940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1769.294729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1769.367351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1769.423554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1769.466688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1769.491677] LNet: 88805:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1769.535463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1769.612872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1769.672888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1769.732513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1769.824873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1769.890518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1770.678296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp -m GET -e local_aborted -r 2 [ 1770.704727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp1 -m GET -e local_aborted -r 2 [ 1770.753818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp -m GET -e local_aborted -r 2 [ 1770.797722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp1 -m GET -e local_aborted -r 2 [ 1770.845521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp1 -d 192.168.203.136@tcp -m GET -e local_aborted -r 2 [ 1770.892278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp1 -d 192.168.203.136@tcp1 -m GET -e local_aborted -r 2 [ 1770.978359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp1 -d 192.168.203.36@tcp -m GET -e local_aborted -r 2 [ 1771.009458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp1 -d 192.168.203.36@tcp1 -m GET -e local_aborted -r 2 [ 1771.039405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.36@tcp 192.168.203.136@tcp [ 1771.060241] LNet: There was an unexpected network error while writing to 192.168.203.136: rc = -22 [ 1771.316521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1771.366972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1771.446633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1771.501923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1771.568497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1771.627484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1771.680899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1771.710788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1771.753215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1771.804868] LNet: 88927:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1771.851227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1771.922983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1771.984508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1772.012787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1772.090249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1772.161368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1772.870390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp -m GET -e local_no_route -r 2 [ 1772.929609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp1 -m GET -e local_no_route -r 2 [ 1772.999458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp -m GET -e local_no_route -r 2 [ 1773.048910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp1 -m GET -e local_no_route -r 2 [ 1773.090550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp1 -d 192.168.203.136@tcp -m GET -e local_no_route -r 2 [ 1773.138829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp1 -d 192.168.203.136@tcp1 -m GET -e local_no_route -r 2 [ 1773.185580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp1 -d 192.168.203.36@tcp -m GET -e local_no_route -r 2 [ 1773.227306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp1 -d 192.168.203.36@tcp1 -m GET -e local_no_route -r 2 [ 1773.269791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.36@tcp 192.168.203.136@tcp [ 1773.291746] LNet: There was an unexpected network error while writing to 192.168.203.136: rc = -22 [ 1773.709308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1773.749942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1773.805646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1773.859985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1773.916115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1773.973832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1774.021305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1774.052030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1774.088888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1774.112090] LNet: 89050:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1774.141843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1774.174098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1774.200248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1774.227630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1774.275676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1774.316579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1775.091730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp -m GET -e local_timeout -r 2 [ 1775.121469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp1 -m GET -e local_timeout -r 2 [ 1775.178460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp -m GET -e local_timeout -r 2 [ 1775.226570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp1 -m GET -e local_timeout -r 2 [ 1775.288672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp1 -d 192.168.203.136@tcp -m GET -e local_timeout -r 2 [ 1775.355449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp1 -d 192.168.203.136@tcp1 -m GET -e local_timeout -r 2 [ 1775.441340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp1 -d 192.168.203.36@tcp -m GET -e local_timeout -r 2 [ 1775.511980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp1 -d 192.168.203.36@tcp1 -m GET -e local_timeout -r 2 [ 1775.558463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.36@tcp 192.168.203.136@tcp [ 1775.945332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1775.994270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1776.036808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1776.107786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1776.149552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1776.181605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1776.227610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1776.284702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1776.399164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1776.501357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1776.558932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1776.618451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1776.680919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1776.779856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1776.844989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1777.638680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp -m GET -e remote_dropped -r 2 [ 1777.706475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp -d 192.168.203.136@tcp1 -m GET -e remote_dropped -r 2 [ 1777.744497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp -m GET -e remote_dropped -r 2 [ 1777.811496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp -d 192.168.203.36@tcp1 -m GET -e remote_dropped -r 2 [ 1777.908114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp1 -d 192.168.203.136@tcp -m GET -e remote_dropped -r 2 [ 1778.012396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.36@tcp1 -d 192.168.203.136@tcp1 -m GET -e remote_dropped -r 2 [ 1778.075396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp1 -d 192.168.203.36@tcp -m GET -e remote_dropped -r 2 [ 1778.131231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.36@tcp1 -d 192.168.203.36@tcp1 -m GET -e remote_dropped -r 2 [ 1778.256540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.36@tcp 192.168.203.136@tcp [ 1778.319788] LNet: There was an unexpected network error while writing to 192.168.203.136: rc = -22 [ 1778.334508] LNet: Skipped 1 previous similar message [ 1778.341423] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.136@tcp [ 1778.682729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1778.741891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1778.794781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1778.828982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1778.889686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1778.945440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1778.995544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1779.030484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1779.061578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1779.086892] LNet: 89293:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1779.098457] LNet: 89293:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1779.132773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1779.177957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1779.217309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1779.265247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1779.327952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1779.360051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1784.725763] LNet: 89368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1784.735362] LNetError: 89368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1784.741514] LNet: Removed LNI 192.168.203.36@tcp [ 1784.764955] LNet: Removed LNI 192.168.203.36@tcp1 [ 1785.219663] Key type .llcrypt unregistered [ 1785.225782] Key type ._llcrypt unregistered [ 1802.795769] Key type ._llcrypt registered [ 1802.798632] Key type .llcrypt registered [ 1804.030406] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 17:40:10 (1788644410) [ 1804.758925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1804.786945] alg: No test for adler32 (adler32-zlib) [ 1805.717540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1805.779704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1805.801918] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1806.619565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1806.649601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.36@tcp [ 1806.665763] LNetError: 90344:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.36@tcp to recovery [ 1806.740767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.36@tcp [ 1806.847978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1812.867919] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 17:40:19 (1788644419) [ 1813.184914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1813.214900] LNet: 90934:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1813.224899] LNetError: 90934:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1813.238527] LNet: Removed LNI 192.168.203.36@tcp [ 1813.294910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1813.369896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1813.391486] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1814.031703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1814.065694] LNet: Added LNI 192.168.203.36@tcp1 [8/256/0/180] [ 1815.005811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.36@tcp [ 1815.042879] LNetError: 91201:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.36@tcp to recovery [ 1815.391371] LNet: 1 local NIs in recovery (showing 1): 192.168.203.36@tcp [ 1815.393793] LNet: There was an unexpected network error while writing to 192.168.203.36: rc = -22 [ 1816.314476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.36@tcp [ 1824.524973] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 17:40:31 (1788644431) [ 1826.195840] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg336-server [ 1827.896826] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 17:40:34 (1788644434) [ 1829.754078] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg336-server [ 1831.823685] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 17:40:38 (1788644438) [ 1833.554886] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg336-server [ 1835.447270] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 17:40:42 (1788644442) [ 1835.616516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1835.656146] LNet: 92816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1835.666407] LNetError: 92816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1835.676553] LNet: Removed LNI 192.168.203.36@tcp [ 1835.720502] LNet: Removed LNI 192.168.203.36@tcp1 [ 1835.751454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1835.865155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1835.919991] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1836.857941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1836.957755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1837.035670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1837.184233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1837.232412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1837.312355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1843.495816] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 17:40:50 (1788644450) [ 1843.635953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1843.675126] LNet: 93546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1843.687540] LNetError: 93546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1843.699848] LNetError: 93546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1843.722216] LNet: Removed LNI 192.168.203.36@tcp [ 1843.757037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1843.859488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1843.901566] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1851.176387] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 17:40:57 (1788644457) [ 1851.495650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1851.511268] LNet: 94256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1851.515771] LNetError: 94256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1851.534515] LNet: Removed LNI 192.168.203.36@tcp [ 1851.552062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1851.601447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1851.661197] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1858.868635] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 17:41:05 (1788644465) [ 1860.299704] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1862.329701] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 17:41:09 (1788644469) [ 1862.813364] LNet: 95291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1862.822029] LNetError: 95291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1862.839828] LNet: Removed LNI 192.168.203.36@tcp [ 1863.705933] Key type .llcrypt unregistered [ 1863.708515] Key type ._llcrypt unregistered [ 1870.433624] Key type ._llcrypt registered [ 1870.435808] Key type .llcrypt registered [ 1870.678729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1870.694663] alg: No test for adler32 (adler32-zlib) [ 1871.591759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1871.642467] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1878.487682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.136@tcp [ 1883.779761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.136@tcp [ 1885.205768] LNet: 95707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1885.216394] LNetError: 95707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1885.242306] LNet: Removed LNI 192.168.203.36@tcp [ 1885.790767] Key type .llcrypt unregistered [ 1885.793171] Key type ._llcrypt unregistered [ 1896.801277] Key type ._llcrypt registered [ 1896.808841] Key type .llcrypt registered [ 1898.158808] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1899.974239] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1902.101882] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 17:41:48 (1788644508) [ 1903.739175] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1906.007601] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 17:41:52 (1788644512) [ 1908.168713] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg336-server [ 1910.511588] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 17:41:56 (1788644516) [ 1913.215581] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg336-server [ 1915.590432] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 17:42:01 (1788644521) [ 1917.478450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1917.602467] alg: No test for adler32 (adler32-zlib) [ 1918.808796] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 1919.142614] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1920.839610] Key type lgssc registered [ 1922.819880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1945.206242] Key type lgssc unregistered [ 1945.532598] LNet: 98894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1945.563894] LNetError: 98894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1945.599770] LNet: Removed LNI 192.168.203.36@tcp [ 1946.288503] Key type .llcrypt unregistered [ 1946.291984] Key type ._llcrypt unregistered [ 1962.341674] Key type ._llcrypt registered [ 1962.343434] Key type .llcrypt registered [ 1964.010606] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 17:42:50 (1788644570) [ 1964.456992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1964.476961] alg: No test for adler32 (adler32-zlib) [ 1965.478573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1965.571795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1965.625429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1965.679205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1965.731605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1972.099867] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 17:42:58 (1788644578) [ 1972.263232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1972.297065] LNet: 100302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1972.335400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1972.420428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1972.462589] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1973.423245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1973.447835] LNet: 100451:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1973.461617] LNetError: 100451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1973.478583] LNet: Removed LNI 192.168.203.36@tcp [ 1973.502757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1973.557232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1973.578981] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1974.174349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1974.206360] LNetError: 100479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1974.223830] LNet: Removed LNI 192.168.203.36@tcp [ 1974.251483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1974.327163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1974.351779] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1982.088927] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 17:43:08 (1788644588) [ 1982.241302] LNet: 101066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1982.246050] LNet: 101066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1982.252598] LNetError: 101066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1982.270425] LNet: Removed LNI 192.168.203.36@tcp [ 1983.147756] Key type .llcrypt unregistered [ 1983.149634] Key type ._llcrypt unregistered [ 1990.365498] Key type ._llcrypt registered [ 1990.376852] Key type .llcrypt registered [ 1990.556206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1990.565518] alg: No test for adler32 (adler32-zlib) [ 1991.574225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1991.687030] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 1991.689504] LNet: *** cfs_fail_loc=e003, val=100*** [ 1992.568417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 1992.602035] LNet: *** cfs_fail_loc=e003, val=100*** [ 1992.613477] LNet: Skipped 2 previous similar messages [ 1992.623117] Ping reply received with mapping: [ 1992.624687] found metadata reply NI[192.168.203.36@tcp] [ 1992.629340] 24 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1992.637744] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1992.642551] 5a 5a 5a 5a 5a 5a 5a 5a [ 1992.823370] LNet: 101464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1992.834495] LNetError: 101464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1992.861806] LNet: Removed LNI 192.168.203.36@tcp [ 1993.536912] Key type .llcrypt unregistered [ 1993.539869] Key type ._llcrypt unregistered [ 2009.447919] Key type ._llcrypt registered [ 2009.451927] Key type .llcrypt registered [ 2011.334762] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 17:43:37 (1788644617) [ 2011.979538] Key type .llcrypt unregistered [ 2011.981412] Key type ._llcrypt unregistered [ 2018.410960] Key type ._llcrypt registered [ 2018.414894] Key type .llcrypt registered [ 2018.709388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2018.723159] alg: No test for adler32 (adler32-zlib) [ 2019.984991] LNetError: 102500:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2020.375843] Key type .llcrypt unregistered [ 2020.379627] Key type ._llcrypt unregistered [ 2025.739044] Key type ._llcrypt registered [ 2025.741996] Key type .llcrypt registered [ 2027.271440] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 17:43:53 (1788644633) [ 2027.942704] Key type .llcrypt unregistered [ 2027.945412] Key type ._llcrypt unregistered [ 2035.094096] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2035.116063] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2037.892300] Key type ._llcrypt registered [ 2037.901550] Key type .llcrypt registered [ 2038.232972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2038.255689] alg: No test for adler32 (adler32-zlib) [ 2039.158320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2039.214392] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2049.126061] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 17:44:15 (1788644655) [ 2049.208559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2049.239458] LNet: 104265:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2049.246778] LNetError: 104265:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2049.266447] LNet: Removed LNI 10.1.2.3@tcp [ 2049.292849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2050.100118] Key type .llcrypt unregistered [ 2050.104214] Key type ._llcrypt unregistered [ 2056.690210] Key type ._llcrypt registered [ 2056.691958] Key type .llcrypt registered [ 2057.026963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2057.052770] alg: No test for adler32 (adler32-zlib) [ 2057.985121] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 2058.052412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 2058.192704] LNet: 104559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2058.199149] LNetError: 104559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2059.241428] LNet: Removed LNI 192.168.203.36@tcp [ 2059.966187] Key type .llcrypt unregistered [ 2059.973488] Key type ._llcrypt unregistered [ 2071.246409] Key type ._llcrypt registered [ 2071.252990] Key type .llcrypt registered [ 2072.870050] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 17:44:39 (1788644679) [ 2073.508254] Key type .llcrypt unregistered [ 2073.510059] Key type ._llcrypt unregistered [ 2079.275957] Key type ._llcrypt registered [ 2079.279500] Key type .llcrypt registered [ 2079.531501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2079.555539] alg: No test for adler32 (adler32-zlib) [ 2086.231923] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 17:44:53 (1788644693) [ 2086.348411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2086.396608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2086.474877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2086.499978] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 2087.250133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2087.293716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2087.339556] LNet: Removed LNI 192.168.203.36@tcp [ 2087.571987] LNet: 106297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2087.924400] Key type .llcrypt unregistered [ 2087.928534] Key type ._llcrypt unregistered [ 2097.618121] Key type ._llcrypt registered [ 2097.622444] Key type .llcrypt registered [ 2098.940645] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 17:45:05 (1788644705) [ 2099.199867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2099.220380] alg: No test for adler32 (adler32-zlib) [ 2100.116680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2100.170457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2100.190506] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 2100.786635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 2100.814616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.36@tcp [ 2100.827379] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2100.830557] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2100.832868] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2100, 0, 0 [ 2100.837583] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2100.839922] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2100.843230] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2100.845652] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2100, 0, 0 [ 2100.850164] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2100.852347] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2100.855689] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2100.858239] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2100, 0, 2150 [ 2100.862429] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2100.864685] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2100.867697] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2100.870747] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2100, 0, 2150 [ 2100.878186] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2106.351929] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 17:45:13 (1788644713) [ 2106.730324] LNet: 107833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2106.736324] LNetError: 107833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2106.751742] LNet: Removed LNI 192.168.203.36@tcp [ 2107.371786] Key type .llcrypt unregistered [ 2107.378661] Key type ._llcrypt unregistered [ 2112.541584] Key type ._llcrypt registered [ 2112.545780] Key type .llcrypt registered [ 2112.830495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2112.860364] alg: No test for adler32 (adler32-zlib) [ 2113.811536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2113.838102] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 2121.600991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.136@tcp [ 2130.206831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2130.282250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2130.326392] LNet: Added LNI 192.168.203.36@tcp1 [8/256/0/180] [ 2131.201873] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2131.361262] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2131.373401] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2131.383926] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2133.880268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2133.908026] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2147.317814] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 17:45:53 (1788644753) [ 2147.630826] LNet: 109334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2147.646408] LNetError: 109334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2147.657548] LNet: Removed LNI 192.168.203.36@tcp [ 2148.704041] LNet: Removed LNI 192.168.203.36@tcp1 [ 2148.713287] LNetError: 109334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2148.725968] LNetError: 109334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2148.742443] LNet: Removed LNI 10.1.2.3@tcp99 [ 2149.401064] Key type .llcrypt unregistered [ 2149.403170] Key type ._llcrypt unregistered [ 2154.749506] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2154.756373] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2157.358320] Key type ._llcrypt registered [ 2157.364379] Key type .llcrypt registered [ 2157.572881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2157.629054] alg: No test for adler32 (adler32-zlib) [ 2158.574719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2158.623304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2158.642979] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 2159.432224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2159.475606] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2162.189927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2162.280829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2162.348227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2162.383289] LNetError: 110075:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2162.454039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2162.497465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.36@tcp [ 2163.236203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2163.348521] LNet: 110095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2163.366273] LNetError: 110095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2163.378274] LNet: Removed LNI 10.1.2.3@tcp [ 2164.390070] LNet: Removed LNI 192.168.203.36@tcp [ 2165.288686] Key type .llcrypt unregistered [ 2165.295139] Key type ._llcrypt unregistered [ 2181.605839] Key type ._llcrypt registered [ 2181.607970] Key type .llcrypt registered [ 2183.456635] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 17:46:29 (1788644789) [ 2183.914521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2183.949237] alg: No test for adler32 (adler32-zlib) [ 2184.946525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2184.986943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2185.006148] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 2185.720906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 2185.830061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg336-client@tcp [ 2193.002322] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 17:46:39 (1788644799) [ 2193.114036] LNet: 111616:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2193.125862] LNetError: 111616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2193.172138] LNet: Removed LNI 192.168.203.36@tcp [ 2193.919245] Key type .llcrypt unregistered [ 2193.921775] Key type ._llcrypt unregistered [ 2199.393251] Key type ._llcrypt registered [ 2199.402284] Key type .llcrypt registered [ 2199.745868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2199.778622] alg: No test for adler32 (adler32-zlib) [ 2200.765560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2200.812628] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 2200.863808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2207.920439] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 17:46:54 (1788644814) [ 2209.922733] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2212.308365] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 17:46:58 (1788644818) [ 2213.971671] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2216.049773] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 17:47:02 (1788644822) [ 2216.140410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2216.174904] LNet: 113042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2216.191776] LNetError: 113042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2216.217409] LNet: Removed LNI 192.168.203.36@tcp [ 2216.279171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2216.374196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2216.405902] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 2217.045643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2217.104974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2224.308336] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 17:47:10 (1788644830) [ 2224.400829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2224.420254] LNet: 113749:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2224.425890] LNetError: 113749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2224.439751] LNet: Removed LNI 192.168.203.36@tcp [ 2224.474403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2224.572929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2224.620798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2225.467955] Key type .llcrypt unregistered [ 2225.469882] Key type ._llcrypt unregistered [ 2237.275932] Key type ._llcrypt registered [ 2237.278646] Key type .llcrypt registered [ 2238.594869] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 17:47:25 (1788644845) [ 2238.960673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2238.975814] alg: No test for adler32 (adler32-zlib) [ 2239.970354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2240.058314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2240.083104] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 2240.954615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2241.037252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.36@tcp [ 2247.462528] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 17:47:34 (1788644854) [ 2247.545364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2247.571525] LNet: 115303:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2247.586319] LNetError: 115303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2248.618528] LNet: Removed LNI 192.168.203.36@tcp [ 2248.652678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2248.743524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2248.790370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2254.762166] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 17:47:41 (1788644861) [ 2254.894083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2254.931935] LNet: 115888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2254.990641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2255.103070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2255.615725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2255.648584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2255.729785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2256.183846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2256.217988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2256.283932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2256.300566] LNet: 115923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2256.335187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2256.466168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2256.819525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2256.852494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2256.931472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2257.385653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2257.411734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2263.299149] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 17:47:50 (1788644870) [ 2263.397585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2263.426881] LNet: 116518:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2263.465233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2263.539144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2263.568207] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 2264.195981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2264.215921] LNet: Added LNI 192.168.203.36@tcp2 [8/256/0/180] [ 2264.971090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2265.030321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.36@tcp [ 2265.274408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2265.291504] LNetError: 116802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2265.301426] LNet: Removed LNI 192.168.203.36@tcp [ 2265.324911] LNet: Removed LNI 192.168.203.36@tcp2 [ 2265.390327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2265.507782] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 2265.513731] LNet: Added LNI 192.168.203.36@tcp2 [8/256/0/180] [ 2265.520800] LNet: 116808:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2266.351700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.36@tcp [ 2266.584648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2266.620978] LNetError: 116846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2266.629874] LNetError: 116846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2266.644876] LNet: Removed LNI 192.168.203.36@tcp [ 2266.660622] LNet: Removed LNI 192.168.203.36@tcp2 [ 2266.692263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2266.751268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2266.775894] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 2267.300990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2267.315708] LNet: Added LNI 192.168.203.36@tcp2 [8/256/0/180] [ 2268.122960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.36@tcp [ 2268.174378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2268.255628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.36@tcp [ 2268.546485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2268.568342] LNet: 117130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2268.574870] LNet: 117130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2269.599154] LNet: Removed LNI 192.168.203.36@tcp [ 2269.608887] LNetError: 117130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2269.629139] LNetError: 117130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2269.643895] LNet: Removed LNI 192.168.203.36@tcp2 [ 2269.676485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2269.781133] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 2269.791962] LNet: Added LNI 192.168.203.36@tcp2 [8/256/0/180] [ 2269.800682] LNet: 117136:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2270.584910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.36@tcp [ 2270.845635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2270.870364] LNet: Removed LNI 192.168.203.36@tcp [ 2271.909753] LNet: Removed LNI 192.168.203.36@tcp2 [ 2271.935580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2272.030556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2272.082807] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 2272.808854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2272.823437] LNet: Added LNI 192.168.203.36@tcp2 [8/256/0/180] [ 2273.716310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2273.811986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.36@tcp [ 2274.135490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2274.172252] LNetError: 117453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2274.184066] LNetError: 117453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2274.195749] LNet: Removed LNI 192.168.203.36@tcp [ 2274.211809] LNet: Removed LNI 192.168.203.36@tcp2 [ 2274.259275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2274.320436] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 2274.328730] LNet: Added LNI 192.168.203.36@tcp2 [8/256/0/180] [ 2274.333847] LNet: 117459:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2274.871955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.36@tcp [ 2275.082255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2276.127597] LNet: Removed LNI 192.168.203.36@tcp [ 2276.150084] LNet: Removed LNI 192.168.203.36@tcp2 [ 2276.207483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2276.279923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2276.307811] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 2277.153434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2277.186845] LNet: Added LNI 192.168.203.36@tcp2 [8/256/0/180] [ 2278.143282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.36@tcp [ 2278.184250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2278.234227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.36@tcp [ 2278.467371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2278.484321] LNet: 117780:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2278.488941] LNet: 117780:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2279.519194] LNet: Removed LNI 192.168.203.36@tcp [ 2280.554603] LNet: Removed LNI 192.168.203.36@tcp2 [ 2280.640991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2280.733215] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 2280.751567] LNet: Added LNI 192.168.203.36@tcp2 [8/256/0/180] [ 2280.761843] LNet: 117787:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2281.293968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.36@tcp [ 2281.508258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2281.530590] LNet: Removed LNI 192.168.203.36@tcp [ 2282.602881] LNet: Removed LNI 192.168.203.36@tcp2 [ 2282.649253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2282.725506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2282.756854] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 2283.404901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2283.420157] LNet: Added LNI 192.168.203.36@tcp2 [8/256/0/180] [ 2284.133710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2284.166633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2284.229171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.36@tcp [ 2284.542968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2284.567847] LNetError: 118120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2284.573493] LNetError: 118120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2284.580010] LNet: Removed LNI 192.168.203.36@tcp [ 2284.591417] LNet: Removed LNI 192.168.203.36@tcp2 [ 2284.615509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2284.692043] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 2284.702772] LNet: Added LNI 192.168.203.36@tcp2 [8/256/0/180] [ 2285.306644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.36@tcp [ 2285.683402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2286.751749] LNet: Removed LNI 192.168.203.36@tcp [ 2286.773660] LNet: Removed LNI 192.168.203.36@tcp2 [ 2286.804303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2286.879044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2286.921482] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 2287.658980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2287.681446] LNet: Added LNI 192.168.203.36@tcp2 [8/256/0/180] [ 2288.521693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.36@tcp [ 2288.554419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2288.585484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2288.631378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.36@tcp [ 2288.988726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2290.079220] LNet: Removed LNI 192.168.203.36@tcp [ 2291.115638] LNet: Removed LNI 192.168.203.36@tcp2 [ 2291.151800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2291.226180] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 2291.234039] LNet: Added LNI 192.168.203.36@tcp2 [8/256/0/180] [ 2291.246262] LNet: 118486:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2291.252619] LNet: 118486:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 2291.867945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.36@tcp [ 2300.341072] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2301.959878] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 17:48:28 (1788644908) [ 2302.083672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2302.143830] LNet: 119238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2302.153341] LNet: 119238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2302.174627] LNetError: 119238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2302.184882] LNetError: 119238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2302.198271] LNet: Removed LNI 192.168.203.36@tcp [ 2303.276631] LNet: Removed LNI 192.168.203.36@tcp2 [ 2303.317876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2303.709517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2303.729415] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 2304.415179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2304.442422] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2305.279317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2305.316044] LNet: Removed LNI 10.1.2.3@tcp [ 2305.333172] LNet: Removed LNI 192.168.203.36@tcp [ 2305.354447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2305.454105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2305.498358] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 2305.507455] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2305.903919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2305.938220] LNet: Removed LNI 192.168.203.36@tcp [ 2305.949319] LNet: Removed LNI 10.1.2.3@tcp [ 2313.348923] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 17:48:39 (1788644919) [ 2313.523780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2313.602293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2313.718035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2313.762043] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 2315.036040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2315.082853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2315.131690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 2315.181467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 2315.265823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 2315.331030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 2315.385811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 2315.442494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 2315.514153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 2315.597328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 2315.664200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 2315.714976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 2315.954177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2323.403833] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 17:48:49 (1788644929) [ 2323.519990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2324.582242] LNet: Removed LNI 192.168.203.36@tcp [ 2324.630983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2324.723974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2324.759224] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 2325.657522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2325.681979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2325.721913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 2325.773201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 2325.813729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 2325.869180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 2325.901587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 2325.940742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 2325.999772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 2326.039763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 2326.075919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 2326.128522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 2326.278910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2326.340825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 2326.371312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 2326.392335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 2326.443940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 2326.497213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 2326.554776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 2326.622149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 2326.659194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 2326.700258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 2326.731978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.36@tcp [ 2332.777974] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 17:48:59 (1788644939) [ 2332.975706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2334.056094] LNet: Removed LNI 192.168.203.36@tcp [ 2334.092696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2334.221537] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2334.306072] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2334.316622] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2334.331737] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2336.896314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2336.943879] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 2337.692295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2337.719520] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2339.773375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.36@tcp [ 2339.828213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2339.903499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.36@tcp -m PUT -e local_timeout -r 1 [ 2339.966968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2340.045959] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2340.049413] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2340.299647] LNetError: 121688:0:(lib-move.c:4812:lnet_parse()) 192.168.203.36@tcp, src 192.168.203.36@tcp: Dropping PUT (error -108 looking up sender) [ 2341.346738] LNet: 122147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2341.355860] LNet: 122147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2341.364686] LNetError: 122147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2341.375630] LNetError: 122147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2341.390761] LNet: Removed LNI 10.1.2.3@tcp [ 2342.442132] LNet: Removed LNI 192.168.203.36@tcp [ 2342.877661] Key type .llcrypt unregistered [ 2342.880726] Key type ._llcrypt unregistered [ 2353.841902] Key type ._llcrypt registered [ 2353.847022] Key type .llcrypt registered [ 2355.305700] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 17:49:21 (1788644961) [ 2355.696121] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2355.720033] alg: No test for adler32 (adler32-zlib) [ 2356.651171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2357.026784] LNet: 122998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2357.638797] Key type .llcrypt unregistered [ 2357.642257] Key type ._llcrypt unregistered [ 2364.050667] Key type ._llcrypt registered [ 2364.054705] Key type .llcrypt registered [ 2364.319601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2364.346384] alg: No test for adler32 (adler32-zlib) [ 2365.427387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2365.467632] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 2372.919642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.136@tcp [ 2373.179750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2373.996111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.136@tcp [ 2374.273540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2374.310783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2374.336250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2374.362675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2374.391845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2374.422718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2374.460863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2374.483711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2374.510541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2374.536759] LNet: 123447:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2374.555915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2374.595680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2374.622826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2374.650194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2374.686954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2374.687226] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.136@tcp [ 2374.724447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2375.420657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2375.695877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2375.736216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2375.770217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2375.811900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2375.846630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2375.878958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2375.915644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2375.953453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2375.996415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2376.025236] LNet: 123566:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2376.063934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2376.097299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2376.130811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2376.176983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2376.232420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2376.264761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2376.982438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.136@tcp [ 2377.238535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2377.269635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2377.302970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2377.333617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2377.376682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2377.430398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2377.463785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2377.530387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2377.576666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2377.630833] LNet: 123686:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2377.657262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2377.732890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2377.783785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2377.849916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2377.925544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2377.951726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2378.716227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2379.057087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2379.113783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2379.159714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2379.220521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2379.283957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2379.351775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2379.401739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2379.469536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2379.513074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2379.579905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2379.629750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2379.672210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2379.707389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2379.775460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2379.836820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2380.222929] LNet: 123851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2380.252317] LNetError: 123851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2380.285214] LNet: Removed LNI 192.168.203.36@tcp [ 2380.696957] Key type .llcrypt unregistered [ 2380.703609] Key type ._llcrypt unregistered [ 2396.595593] Key type ._llcrypt registered [ 2396.597133] Key type .llcrypt registered [ 2397.733817] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 17:50:04 (1788645004) [ 2398.150789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2398.169350] alg: No test for adler32 (adler32-zlib) [ 2399.070918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2399.477437] LNet: 124737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2400.094652] Key type .llcrypt unregistered [ 2400.097954] Key type ._llcrypt unregistered [ 2406.954538] Key type ._llcrypt registered [ 2406.962968] Key type .llcrypt registered [ 2407.268130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2407.283651] alg: No test for adler32 (adler32-zlib) [ 2408.312555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2408.346435] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 2414.952278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.136@tcp [ 2422.064649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2422.109427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2422.125654] LNet: Added LNI 192.168.203.36@tcp1 [8/256/0/180] [ 2423.589771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2424.532718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.136@tcp [ 2424.735702] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.136@tcp [ 2424.932909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2424.987695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2425.036966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2425.071446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2425.124312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2425.170311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2425.229187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2425.280432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2425.330886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2425.370921] LNet: 125387:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2425.398415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2425.452889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2425.542449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2425.614806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2425.672860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2425.730447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2426.816779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2427.192980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2427.222827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2427.289681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2427.388536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2427.464550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2427.576325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2427.665551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2427.775325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2427.873306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2427.937471] LNet: 125507:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2427.974058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2428.049388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2428.139972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2428.218803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2428.291080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2428.382664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2429.613533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.136@tcp [ 2430.083546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2430.162923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2430.209066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2430.293120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2430.359877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2430.442031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2430.518756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2430.608841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2430.660406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2430.688764] LNet: 125626:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2430.722890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2430.771919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2430.814502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2430.847469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2430.896555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2430.969213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2432.013617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2432.371317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2432.418812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2432.486820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2432.532604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2432.582357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2432.700461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2432.774753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2432.846423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2432.890309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2432.923884] LNet: 125746:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2432.945924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2432.991869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2433.055165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2433.118284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2433.196497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2433.291969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2442.516212] LNet: 125827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2442.527135] LNetError: 125827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2442.537722] LNet: Removed LNI 192.168.203.36@tcp [ 2442.559311] LNet: Removed LNI 192.168.203.36@tcp1 [ 2443.001995] Key type .llcrypt unregistered [ 2443.006165] Key type ._llcrypt unregistered [ 2457.858575] Key type ._llcrypt registered [ 2457.861254] Key type .llcrypt registered [ 2458.984779] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 17:51:05 (1788645065) [ 2459.337340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2459.351817] alg: No test for adler32 (adler32-zlib) [ 2460.296739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2460.361655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2460.391698] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 2461.015983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2461.038293] LNet: Added LNI 192.168.203.36@tcp2 [8/256/0/180] [ 2461.795512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.36@tcp [ 2461.864280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2462.943142] LNet: Removed LNI 192.168.203.36@tcp [ 2468.745517] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 17:51:15 (1788645075) [ 2470.169444] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2471.811500] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 17:51:18 (1788645078) [ 2478.721208] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 17:51:25 (1788645085) [ 2485.011438] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 17:51:31 (1788645091) [ 2491.489835] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 17:51:38 (1788645098) [ 2499.733813] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 17:51:45 (1788645105) [ 2506.961674] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 17:51:53 (1788645113) [ 2514.743717] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 17:52:01 (1788645121) [ 2521.807742] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 17:52:08 (1788645128) [ 2529.313110] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 17:52:15 (1788645135) [ 2535.191593] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 17:52:22 (1788645142) [ 2541.717450] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 17:52:28 (1788645148) [ 2549.913497] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 17:52:36 (1788645156) [ 2556.544260] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 17:52:43 (1788645163) [ 2562.710174] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 17:52:49 (1788645169) [ 2569.595622] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 17:52:56 (1788645176) [ 2576.539235] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 17:53:03 (1788645183) [ 2582.710931] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 17:53:09 (1788645189) [ 2590.469433] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 17:53:17 (1788645197) [ 2590.585882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2590.607325] LNet: 137513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2590.623441] LNetError: 137513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2590.655669] LNet: Removed LNI 192.168.203.36@tcp2 [ 2590.686419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2590.931253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2590.996891] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 2592.295289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2592.350132] LNet: 137662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2592.360077] LNetError: 137662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2592.378791] LNet: Removed LNI 192.168.203.36@tcp [ 2592.404589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2592.478593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2592.505182] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 2599.951469] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 17:53:26 (1788645206) [ 2608.444557] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 17:53:34 (1788645214) [ 2617.486408] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 17:53:43 (1788645223) [ 2625.471872] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 17:53:52 (1788645232) [ 2632.339136] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 17:53:59 (1788645239) [ 2640.035828] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 17:54:06 (1788645246) [ 2647.929746] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 17:54:14 (1788645254) [ 2656.278657] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 17:54:22 (1788645262) [ 2667.475883] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 17:54:33 (1788645273) [ 2676.905975] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 17:54:42 (1788645282) [ 2684.525068] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 17:54:50 (1788645290) [ 2693.140029] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 17:54:59 (1788645299) [ 2700.551369] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2476 sec ========== 17:55:07 (1788645307) [ 2700.862544] LNet: 145128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2700.875616] LNetError: 145128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2700.889416] LNet: Removed LNI 192.168.203.36@tcp [ 2701.655681] Key type .llcrypt unregistered [ 2701.658663] Key type ._llcrypt unregistered [ 2713.698642] Key type ._llcrypt registered [ 2713.714023] Key type .llcrypt registered [ 2714.086983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2714.106241] alg: No test for adler32 (adler32-zlib) [ 2715.280448] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 2715.588163] LNet: Added LNI 192.168.203.36@tcp [8/256/0/180] [ 2717.263713] Key type lgssc registered [ 2718.382561] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2780.915562] Lustre: Mounted lustre-client - version 2.17.58_3_ge47db3b [ 2786.423646] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2795.619118] Lustre: Unmounted lustre-client [ 2819.681856] Key type lgssc unregistered [ 2819.895305] LNet: 148078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2819.914359] LNetError: 148078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2819.943889] LNet: Removed LNI 192.168.203.36@tcp [ 2820.813148] Key type .llcrypt unregistered [ 2820.818308] Key type ._llcrypt unregistered