[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 464165204 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b50-0x000f5b5f] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5970 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003358] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009022] pid_max: default: 32768 minimum: 301 [ 0.010139] LSM: Security Framework initializing [ 0.011062] Yama: becoming mindful. [ 0.012042] SELinux: Initializing. [ 0.013073] *** VALIDATE selinux *** [ 0.021122] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025791] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027118] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029121] *** VALIDATE tmpfs *** [ 0.031373] *** VALIDATE proc *** [ 0.032272] *** VALIDATE cgroup *** [ 0.033020] *** VALIDATE cgroup2 *** [ 0.034254] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035148] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038023] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.042066] debug: unmapping init [mem 0xffffffff9d459000-0xffffffff9d460fff] [ 0.045000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045687] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046023] ... version: 2 [ 0.047014] ... bit width: 48 [ 0.048014] ... generic registers: 4 [ 0.049014] ... value mask: 0000ffffffffffff [ 0.050015] ... max period: 00007fffffffffff [ 0.051015] ... fixed-purpose events: 3 [ 0.052013] ... event mask: 000000070000000f [ 0.053319] rcu: Hierarchical SRCU implementation. [ 0.055510] smp: Bringing up secondary CPUs ... [ 0.056597] x86: Booting SMP configuration: [ 0.057029] .... node #0, CPUs: #1 #2 #3 [ 0.060291] smp: Brought up 1 node, 4 CPUs [ 0.062013] smpboot: Max logical packages: 1 [ 0.063020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.236217] node 0 deferred pages initialised in 168ms [ 0.239149] devtmpfs: initialized [ 0.240254] x86/mm: Memory block size: 128MB [ 0.242803] gcov: version magic: 0x41383552 [ 0.244344] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.245085] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.246253] pinctrl core: initialized pinctrl subsystem [ 0.247244] [ 0.247851] ************************************************************* [ 0.248013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.249014] ** ** [ 0.250015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.251017] ** ** [ 0.252012] ** This means that this kernel is built to expose internal ** [ 0.253016] ** IOMMU data structures, which may compromise security on ** [ 0.254015] ** your system. ** [ 0.255015] ** ** [ 0.256017] ** If you see this message and you are not debugging the ** [ 0.257012] ** kernel, report this immediately to your vendor! ** [ 0.258014] ** ** [ 0.259017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.260016] ************************************************************* [ 0.261693] NET: Registered protocol family 16 [ 0.262442] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.263061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.264069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.265446] cpuidle: using governor menu [ 0.266865] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.269506] PCI: Using configuration type 1 for base access [ 0.272133] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.281024] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.282022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.284036] cryptd: max_cpu_qlen set to 1000 [ 0.286253] ACPI: Added _OSI(Module Device) [ 0.287018] ACPI: Added _OSI(Processor Device) [ 0.288014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.289020] ACPI: Added _OSI(Processor Aggregator Device) [ 0.292507] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.294578] ACPI: Interpreter enabled [ 0.295067] ACPI: PM: (supports S0 S3 S4 S5) [ 0.296019] ACPI: Using IOAPIC for interrupt routing [ 0.297121] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.298387] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.308000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.309071] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.312027] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.315126] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.320454] acpiphp: Slot [2] registered [ 0.323242] acpiphp: Slot [3] registered [ 0.324131] acpiphp: Slot [4] registered [ 0.325138] acpiphp: Slot [5] registered [ 0.326187] acpiphp: Slot [6] registered [ 0.328139] acpiphp: Slot [7] registered [ 0.329090] acpiphp: Slot [8] registered [ 0.330106] acpiphp: Slot [9] registered [ 0.332399] acpiphp: Slot [10] registered [ 0.334184] acpiphp: Slot [11] registered [ 0.336152] acpiphp: Slot [12] registered [ 0.337110] acpiphp: Slot [13] registered [ 0.339109] acpiphp: Slot [14] registered [ 0.340265] acpiphp: Slot [15] registered [ 0.342148] acpiphp: Slot [16] registered [ 0.344160] acpiphp: Slot [17] registered [ 0.346201] acpiphp: Slot [18] registered [ 0.347130] acpiphp: Slot [19] registered [ 0.349130] acpiphp: Slot [20] registered [ 0.350157] acpiphp: Slot [21] registered [ 0.352155] acpiphp: Slot [22] registered [ 0.354194] acpiphp: Slot [23] registered [ 0.355098] acpiphp: Slot [24] registered [ 0.356123] acpiphp: Slot [25] registered [ 0.358134] acpiphp: Slot [26] registered [ 0.359168] acpiphp: Slot [27] registered [ 0.361132] acpiphp: Slot [28] registered [ 0.362132] acpiphp: Slot [29] registered [ 0.364106] acpiphp: Slot [30] registered [ 0.365140] acpiphp: Slot [31] registered [ 0.367117] PCI host bridge to bus 0000:00 [ 0.368024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.370042] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.372030] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.375048] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.378030] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.381028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.383226] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.388536] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.392690] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.400490] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.404997] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.407019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.409019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.411016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.414393] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.416802] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.419067] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.422702] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.427000] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.437742] pci 0000:00:02.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.441019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.449078] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.454023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.460027] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.472021] pci 0000:00:05.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.486431] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.494015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.498020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.513026] pci 0000:00:06.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.525023] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.528419] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.530457] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.532423] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.535317] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.539129] iommu: Default domain type: Passthrough [ 0.541426] SCSI subsystem initialized [ 0.542138] ACPI: bus type USB registered [ 0.544114] usbcore: registered new interface driver usbfs [ 0.546091] usbcore: registered new interface driver hub [ 0.547078] usbcore: registered new device driver usb [ 0.549161] pps_core: LinuxPPS API ver. 1 registered [ 0.551015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.555067] PTP clock support registered [ 0.560178] EDAC MC: Ver: 3.0.0 [ 0.562178] PCI: Using ACPI for IRQ routing [ 0.564965] NetLabel: Initializing [ 0.566018] NetLabel: domain hash size = 128 [ 0.568018] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.570146] NetLabel: unlabeled traffic allowed by default [ 0.572186] vgaarb: loaded [ 0.574317] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.576017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.583373] clocksource: Switched to clocksource kvm-clock [ 0.694603] VFS: Disk quotas dquot_6.6.0 [ 0.696311] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.699038] *** VALIDATE ramfs *** [ 0.700418] *** VALIDATE hugetlbfs *** [ 0.701962] pnp: PnP ACPI init [ 0.704483] pnp: PnP ACPI: found 6 devices [ 0.725161] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.728303] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.730644] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.733027] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.735743] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.738206] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.741224] NET: Registered protocol family 2 [ 0.744156] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.751555] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.755460] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.759841] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.763989] TCP: Hash tables configured (established 65536 bind 65536) [ 0.766904] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.769961] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.772167] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.775051] NET: Registered protocol family 1 [ 0.777810] RPC: Registered named UNIX socket transport module. [ 0.780286] RPC: Registered udp transport module. [ 0.781950] RPC: Registered tcp transport module. [ 0.783874] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.786175] NET: Registered protocol family 44 [ 0.787812] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.790237] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.792568] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.794853] PCI: CLS 0 bytes, default 64 [ 0.796352] Unpacking initramfs... [ 2.278170] debug: unmapping init [mem 0xffff91847cc64000-0xffff91847ffcffff] [ 2.281891] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.284033] software IO TLB: mapped [mem 0x00000000a1c00000-0x00000000a5c00000] (64MB) [ 2.286928] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.793469] Initialise system trusted keyrings [ 2.795375] Key type blacklist registered [ 2.797527] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.810466] zbud: loaded [ 2.813775] *** VALIDATE nfs *** [ 2.815073] *** VALIDATE nfs4 *** [ 2.816625] pstore: using deflate compression [ 2.820994] Platform Keyring initialized [ 2.947143] NET: Registered protocol family 38 [ 2.949048] Key type asymmetric registered [ 2.950400] Asymmetric key parser 'x509' registered [ 2.952413] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.955289] io scheduler mq-deadline registered [ 2.957153] io scheduler kyber registered [ 2.959109] io scheduler bfq registered [ 2.960833] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.966527] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.969520] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.972765] ACPI: Power Button [PWRF] [ 3.069848] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.164675] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.272601] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.302015] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.333564] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.339370] Non-volatile memory driver v1.3 [ 3.341180] Linux agpgart interface v0.103 [ 3.377129] virtio_blk virtio1: [vda] 133848 512-byte logical blocks (68.5 MB/65.4 MiB) [ 3.381217] vda: detected capacity change from 0 to 68530176 [ 3.397768] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.401868] vdb: detected capacity change from 0 to 1073741824 [ 3.409762] libphy: Fixed MDIO Bus: probed [ 3.419233] usbcore: registered new interface driver usbserial_generic [ 3.421806] usbserial: USB Serial support registered for generic [ 3.424224] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.428515] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.430231] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.432723] mousedev: PS/2 mouse device common for all mice [ 3.435911] rtc_cmos 00:05: RTC can wake from S4 [ 3.440856] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.442317] rtc_cmos 00:05: registered as rtc0 [ 3.447065] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.448821] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.450798] intel_pstate: CPU model not supported [ 3.456809] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.462158] hid: raw HID events driver (C) Jiri Kosina [ 3.464209] usbcore: registered new interface driver usbhid [ 3.466129] usbhid: USB HID core driver [ 3.467730] drop_monitor: Initializing network drop monitor service [ 3.469619] Initializing XFRM netlink socket [ 3.471090] NET: Registered protocol family 10 [ 3.475360] Segment Routing with IPv6 [ 3.476849] NET: Registered protocol family 17 [ 3.479149] mpls_gso: MPLS GSO support [ 3.488897] RAS: Correctable Errors collector initialized. [ 3.491322] AVX version of gcm_enc/dec engaged. [ 3.493249] AES CTR mode by8 optimization enabled [ 3.575143] sched_clock: Marking stable (3575119327, 0)->(4543675304, -968555977) [ 3.579246] registered taskstats version 1 [ 3.585102] Loading compiled-in X.509 certificates [ 3.587479] zswap: loaded using pool lzo/zbud [ 3.614122] Key type big_key registered [ 3.628796] Key type encrypted registered [ 3.630362] ima: No TPM chip found, activating TPM-bypass! [ 3.632523] ima: Allocated hash algorithm: sha1 [ 3.634489] ima: No architecture policies found [ 3.636489] evm: Initialising EVM extended attributes: [ 3.638591] evm: security.selinux [ 3.639966] evm: security.ima [ 3.640949] evm: security.capability [ 3.642302] evm: HMAC attrs: 0x1 [ 3.644669] rtc_cmos 00:05: setting system clock to 2025-11-16 22:57:51 UTC (1763333871) [ 3.651735] debug: unmapping init [mem 0xffffffff9e403000-0xffffffff9e5fffff] [ 3.655062] debug: unmapping init [mem 0xffffffff9d182000-0xffffffff9d458fff] [ 3.663154] Write protecting the kernel read-only data: 28672k [ 3.667738] debug: unmapping init [mem 0xffffffff9b803000-0xffffffff9b9fffff] [ 3.670695] debug: unmapping init [mem 0xffffffff9c114000-0xffffffff9c1fffff] [ 3.703323] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.711946] systemd[1]: Detected virtualization kvm. [ 3.714219] systemd[1]: Detected architecture x86-64. [ 3.716197] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.745374] systemd[1]: No hostname configured. [ 3.747282] systemd[1]: Set hostname to . [ 3.749241] random: systemd: uninitialized urandom read (16 bytes read) [ 3.751884] systemd[1]: Initializing machine ID from random generator. [ 3.810916] random: ln: uninitialized urandom read (6 bytes read) [ 3.898691] random: systemd: uninitialized urandom read (16 bytes read) [ 3.901743] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.906385] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.911492] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.526158] device-mapper: uevent: version 1.0.3 [ 4.529074] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.329204] virtio_net virtio0 ens2: renamed from eth0 [ 5.363263] scsi host0: ata_piix [ 5.372521] scsi host1: ata_piix [ 5.374216] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.376869] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 7.203739] random: fast init done [ 9.111452] dracut-initqueue[583]: RTNETLINK answers: File exists [ 10.147930] random: crng init done [ 10.149371] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.673464] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.866477] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.141397] SELinux: Disabled at runtime. [ 12.208216] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.216893] systemd[1]: Detected virtualization kvm. [ 12.218498] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.796417] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.801241] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.805679] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.810235] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.815428] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.831462] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.838101] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Startin[ 13.011883] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS g udev Coldplug all Devices... Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.389152] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.831760] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.842440] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.901638] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.923474] EDAC sbridge: Ver: 1.1.2 [ 15.380742] Key type dns_resolver registered [ 15.690167] NFS: Registering the id_resolver key type [ 15.692184] Key type id_resolver registered [ 15.693757] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg332-client login: [ 47.809708] libcfs: loading out-of-tree module taints kernel. [ 47.880765] Key type ._llcrypt registered [ 47.882110] Key type .llcrypt registered [ 48.312887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 48.320946] alg: No test for adler32 (adler32-zlib) [ 49.383602] Lustre: Lustre: Build Version: 2.16.61_54_g3c0e0e6 [ 49.762606] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 51.399180] Key type lgssc registered [ 52.138889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 108.182009] hrtimer: interrupt took 8000835 ns [ 166.080217] Lustre: Mounted lustre-client [ 170.646543] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 185.762463] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing check_logdir /tmp/testlogs/ [ 189.666210] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing yml_node [ 191.969896] Lustre: lustre-OST0000-osc-ffff9184c3062800: disconnect after 24s idle [ 194.750387] Lustre: DEBUG MARKER: Client: 2.16.61.54 [ 196.985990] Lustre: DEBUG MARKER: MDS: 2.16.61.54 [ 199.092564] Lustre: DEBUG MARKER: OSS: 2.16.61.54 [ 200.404950] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sun Nov 16 18:01:07 EST 2025 [ 213.863660] LustreError: 7916:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9184c3062800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 213.871296] LustreError: 7916:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 213.899517] Lustre: Unmounted lustre-client [ 251.081837] Key type lgssc unregistered [ 251.320223] LNet: 8365:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 251.329408] LNetError: 8365:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 251.391231] LNet: Removed LNI 192.168.203.32@tcp [ 252.095051] Key type .llcrypt unregistered [ 252.096498] Key type ._llcrypt unregistered [ 272.783427] Key type ._llcrypt registered [ 272.798817] Key type .llcrypt registered [ 273.069043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 273.095473] alg: No test for adler32 (adler32-zlib) [ 274.059088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 274.166500] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 274.188588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 277.960163] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 278.090601] LNet: 9229:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 278.106037] LNetError: 9229:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 278.137563] LNet: Removed LNI 192.168.203.32@tcp [ 278.748867] Key type .llcrypt unregistered [ 278.750574] Key type ._llcrypt unregistered [ 284.393714] Key type ._llcrypt registered [ 284.398141] Key type .llcrypt registered [ 285.686138] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 18:02:32 (1763334152) [ 286.110712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 286.128561] alg: No test for adler32 (adler32-zlib) [ 287.017334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 287.090337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 287.119970] LNet: 9645:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 287.148119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 287.200056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 287.216993] LNet: 9652:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 293.330270] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 18:02:39 (1763334159) [ 293.427985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 293.461063] LNet: 10223:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 293.501228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 293.606484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 299.562956] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 18:02:46 (1763334166) [ 299.689628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 299.713685] LNet: 10807:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 299.734028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 299.852736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 305.772675] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 18:02:52 (1763334172) [ 305.861117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 305.879590] LNet: 11392:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 305.910282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 306.012185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 311.249894] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 18:02:57 (1763334177) [ 311.311792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 311.333187] LNet: 11974:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 311.367901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 311.500721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 311.856133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 318.330117] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 18:03:04 (1763334184) [ 318.484169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 318.552671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 318.788279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 319.488609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 320.345349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 328.182833] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 18:03:14 (1763334194) [ 328.375662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 328.400053] LNet: 13165:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 328.414510] LNet: 13165:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 328.474943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 328.566579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 335.785277] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 18:03:22 (1763334202) [ 335.945539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 336.026531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 336.120915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 336.150598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 336.529788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 336.569514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 336.960851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 336.993675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 337.408838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 337.474482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 337.841381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 337.879659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 338.290467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 338.333338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 338.744428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 338.791826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 339.369992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 339.423339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 339.942304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 339.992744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 340.331608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 340.356595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 340.636627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 340.679354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 341.008836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 341.053740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 341.469315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 341.499703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 348.096299] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 18:03:34 (1763334214) [ 348.192645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 348.218778] LNet: 14431:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 348.229331] LNet: 14431:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 348.266929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 348.370665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 348.414340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 354.621760] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 18:03:41 (1763334221) [ 354.761503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 354.848770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 354.974861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 355.018854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 360.618715] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 18:03:47 (1763334227) [ 360.856925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 360.903670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 360.995885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 361.024531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 366.606980] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 18:03:53 (1763334233) [ 366.784729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 366.871395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 366.985574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 367.025935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 372.377761] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 18:03:59 (1763334239) [ 372.493198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 372.522784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 372.617504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 372.655947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 378.365724] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 18:04:04 (1763334244) [ 378.490973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 378.527217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 378.602671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 378.631463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 384.982156] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 18:04:11 (1763334251) [ 385.056712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 385.076861] LNet: 17948:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 385.082903] LNet: 17948:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 385.144538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 385.284655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 385.895797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 392.220419] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 18:04:18 (1763334258) [ 392.320530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.348457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.587750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 392.889353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 398.453493] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 18:04:25 (1763334265) [ 398.567484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 398.627806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.759342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 399.001332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 404.047582] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 18:04:30 (1763334270) [ 404.116253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 404.149455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 404.566137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 404.819105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 410.025814] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 18:04:36 (1763334276) [ 410.147397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 410.211268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 410.278207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 410.313970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 415.326442] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 18:04:42 (1763334282) [ 415.509598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 415.560953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 415.629625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 415.670308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 421.465918] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 18:04:48 (1763334288) [ 421.556520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.600109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.673394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 427.432360] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 18:04:54 (1763334294) [ 427.548237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 427.590258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 427.689512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 433.718192] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 18:05:00 (1763334300) [ 433.816699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 433.879937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 433.978776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 434.493021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 440.718663] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 18:05:07 (1763334307) [ 440.873298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 440.957688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 441.091414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 441.129173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 447.152843] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 18:05:13 (1763334313) [ 447.237066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 447.272334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 447.380277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 447.421399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 453.728420] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 18:05:20 (1763334320) [ 453.926951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.958545] LNet: 24565:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 453.967273] LNet: 24565:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 454.008410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.095881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 454.138201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 460.474512] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 18:05:27 (1763334327) [ 460.558204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.607820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.714291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 460.746526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 466.057426] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 18:05:32 (1763334332) [ 466.269814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 466.297671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 466.349181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 466.409106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 466.427933] LNetError: 25751:0:(peer.c:2216:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 466.459318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 466.490989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 471.522304] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 18:05:38 (1763334338) [ 471.775499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 471.820427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 471.881024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Ab8VQo19Vq [ 471.947897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Ab8VQo19Vq [ 471.985320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Ab8VQo19Vq [ 476.332756] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 18:05:43 (1763334343) [ 476.452280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.507898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.558326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 476.585960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 476.619351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 481.984144] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 18:05:48 (1763334348) [ 482.069585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 482.128487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 482.169566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 484.493533] LNet: Message forwarding will be enabled in 2s [ 487.584453] LNet: Message forwarding enabled [ 487.937600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 487.952849] LNet: Message forwarding will stop in 2s to 6s [ 492.403464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 492.989096] LNet: Message forwarding disabled [ 494.586633] LNet: Message forwarding will be enabled in 2s [ 498.144101] LNet: Message forwarding enabled [ 501.655339] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 18:06:08 (1763334368) [ 502.057384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 504.512740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 504.654153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 511.447834] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 18:06:17 (1763334377) [ 511.773918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 511.824968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 511.924474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 511.974503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 511.989876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 512.006180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 512.121963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 512.167469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 512.187944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 512.203325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 512.282302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 512.322762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 512.374415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 512.426913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 512.456864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 512.502550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 512.599180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 512.616041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 512.652101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 512.708889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 512.744427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 512.773856] LNetError: 28820:0:(router.c:629:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 519.828515] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 18:06:26 (1763334386) [ 519.919159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.968035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 520.056664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 525.390315] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 18:06:32 (1763334392) [ 527.066880] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg332-server [ 528.793560] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 18:06:35 (1763334395) [ 529.034126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 529.087986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 529.154353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 529.175905] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 530.626837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[33-37]@tcp [ 530.650395] LNet: 30536:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 530.720783] LNetError: 30326:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.34@tcp is being used as a gateway but routing feature is not turned on [ 530.742031] LNetError: 30326:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.34@tcp has gone from up to down [ 531.688503] LNetError: 30326:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.35@tcp is being used as a gateway but routing feature is not turned on [ 531.702134] LNetError: 30326:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 532.707092] LNetError: 30326:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.33@tcp is being used as a gateway but routing feature is not turned on [ 532.722988] LNetError: 30326:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 534.759115] LNetError: 30326:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.36@tcp is being used as a gateway but routing feature is not turned on [ 534.775984] LNetError: 30326:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 536.524126] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 18:06:43 (1763334403) [ 538.200502] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg332-server [ 538.855341] LNetError: 30326:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.33@tcp is being used as a gateway but routing feature is not turned on [ 538.879595] LNetError: 30326:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 539.831447] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 18:06:46 (1763334406) [ 540.074829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 540.106735] LNetError: 31471:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 540.129749] LNet: Removed LNI 192.168.203.32@tcp [ 540.152840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.205577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 540.230794] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 540.862909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[33-38/2]@tcp [ 540.887649] LNet: 31588:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 540.892220] LNet: 31588:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 540.903851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[33-38/2]@tcp [ 540.912762] LNetError: 31475:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.33@tcp has gone from up to down [ 540.933967] LNetError: 31475:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 546.799405] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 18:06:53 (1763334413) [ 547.011584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.034045] LNetError: 32181:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 548.072505] LNet: Removed LNI 192.168.203.32@tcp [ 548.101737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 548.205738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 548.263586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 548.296589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 548.370125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 548.412731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 548.441869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 548.487104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 548.577540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 548.634706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 548.709225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 548.773630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 548.810693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 548.855374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 548.917035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 548.950642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.033182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 549.098971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.130935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.173905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 549.229820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.254788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.305227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 549.377890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.416148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.465563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 549.522149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.554780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.599932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 549.667605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.709604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.761917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 549.819105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.843788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.917558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 554.897084] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 18:07:01 (1763334421) [ 556.368300] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg332-server [ 557.981068] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 18:07:04 (1763334424) [ 559.595523] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg332-server [ 561.176924] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 18:07:07 (1763334427) [ 561.931343] Key type .llcrypt unregistered [ 561.932671] Key type ._llcrypt unregistered [ 566.495520] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 566.649263] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 566.652838] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 566.664504] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 569.281092] Key type ._llcrypt registered [ 569.282614] Key type .llcrypt registered [ 569.440269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 569.450666] alg: No test for adler32 (adler32-zlib) [ 570.344729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 570.379738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 570.400769] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 570.962600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 570.979617] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 572.734543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 572.756871] LNet: Removed LNI 10.1.2.3@tcp [ 572.886217] LNet: 34296:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 572.889759] LNetError: 34296:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 572.900310] LNet: Removed LNI 192.168.203.32@tcp [ 573.481473] Key type .llcrypt unregistered [ 573.483290] Key type ._llcrypt unregistered [ 582.574887] Key type ._llcrypt registered [ 582.576500] Key type .llcrypt registered [ 583.719419] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 18:07:30 (1763334450) [ 584.251473] Key type .llcrypt unregistered [ 584.253945] Key type ._llcrypt unregistered [ 588.624666] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 588.735660] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 588.759107] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 588.761754] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 591.290545] Key type ._llcrypt registered [ 591.292278] Key type .llcrypt registered [ 591.450580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 591.468563] alg: No test for adler32 (adler32-zlib) [ 592.405403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.500669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 592.538523] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 593.076822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 593.097965] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 595.002891] LNet: 35786:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 595.008365] LNetError: 35786:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 595.018143] LNet: Removed LNI 192.168.203.32@tcp [ 595.026707] LNet: Removed LNI 10.1.2.3@tcp [ 595.695160] Key type .llcrypt unregistered [ 595.697950] Key type ._llcrypt unregistered [ 606.389723] Key type ._llcrypt registered [ 606.392199] Key type .llcrypt registered [ 607.393118] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 18:07:54 (1763334474) [ 607.791244] Key type .llcrypt unregistered [ 607.795766] Key type ._llcrypt unregistered [ 612.463981] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 612.529857] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 612.541355] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 612.549910] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 615.038439] Key type ._llcrypt registered [ 615.040835] Key type .llcrypt registered [ 615.216540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 615.242845] alg: No test for adler32 (adler32-zlib) [ 616.143633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.196936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 616.228288] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 618.157475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 618.180716] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 620.178281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 620.197811] LNet: Removed LNI 10.1.2.3@tcp [ 620.212166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 620.237962] LNet: Removed LNI 10.1.2.31@tcp [ 620.254680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 620.304327] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 620.875481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 620.898506] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 621.797039] LNet: 37700:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 621.816246] LNetError: 37700:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 621.819266] LNet: Removed LNI 10.1.2.3@tcp [ 621.846398] LNet: Removed LNI 10.1.2.31@tcp [ 622.440570] Key type .llcrypt unregistered [ 622.442651] Key type ._llcrypt unregistered [ 631.339506] Key type ._llcrypt registered [ 631.341166] Key type .llcrypt registered [ 632.477540] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 18:08:19 (1763334499) [ 632.787078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 632.806850] alg: No test for adler32 (adler32-zlib) [ 633.741647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 633.819235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 633.875290] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 634.583426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 634.614301] LNet: 38648:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 634.629396] LNetError: 38648:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 634.656690] LNet: Removed LNI 192.168.203.32@tcp [ 634.676706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 634.736259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.32@tcp [ 634.760318] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 635.444780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.32@tcp [ 635.466566] LNet: Removed LNI 192.168.203.32@tcp [ 635.530913] LNet: 38783:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 636.028430] Key type .llcrypt unregistered [ 636.030246] Key type ._llcrypt unregistered [ 643.181504] Key type ._llcrypt registered [ 643.183662] Key type .llcrypt registered [ 644.241813] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 18:08:30 (1763334510) [ 645.760802] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg332-server [ 647.096514] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 18:08:33 (1763334513) [ 647.525535] Key type .llcrypt unregistered [ 647.528499] Key type ._llcrypt unregistered [ 651.555661] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 651.652664] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 651.658756] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 651.661483] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 654.013234] Key type ._llcrypt registered [ 654.014947] Key type .llcrypt registered [ 654.141836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 654.154688] alg: No test for adler32 (adler32-zlib) [ 655.023062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 655.069781] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 655.612568] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 657.070547] LNet: 40604:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 657.075081] LNetError: 40604:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 657.080423] LNet: Removed LNI 192.168.203.32@tcp1 [ 657.088953] LNet: Removed LNI 10.1.2.3@tcp2 [ 657.477127] Key type .llcrypt unregistered [ 657.480255] Key type ._llcrypt unregistered [ 664.718593] Key type ._llcrypt registered [ 664.719755] Key type .llcrypt registered [ 665.640364] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 18:08:52 (1763334532) [ 665.859397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 665.867996] alg: No test for adler32 (adler32-zlib) [ 666.797663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 666.866470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 666.910924] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 667.533986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 667.544705] LNet: 41569:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 667.548437] LNetError: 41569:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 667.573112] LNet: Removed LNI 192.168.203.32@tcp [ 667.600814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 667.645429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 667.664624] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 672.632152] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 18:08:59 (1763334539) [ 672.787861] LNet: 42161:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 672.811338] LNetError: 42161:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 672.823236] LNet: Removed LNI 192.168.203.32@tcp [ 673.284510] Key type .llcrypt unregistered [ 673.286157] Key type ._llcrypt unregistered [ 676.737144] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 676.845385] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 676.877407] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 676.884695] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 679.382463] Key type ._llcrypt registered [ 679.388693] Key type .llcrypt registered [ 679.598864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 679.612136] alg: No test for adler32 (adler32-zlib) [ 680.509325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 680.566646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 680.611670] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 681.171478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 681.186245] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 682.777730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 682.797263] LNet: 42894:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 682.804817] LNetError: 42894:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 682.808543] LNet: Removed LNI 192.168.203.32@tcp [ 682.815947] LNet: Removed LNI 10.1.2.3@tcp [ 682.832342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 682.874163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 682.899621] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 682.906381] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 683.262994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 683.282475] LNet: Removed LNI 192.168.203.32@tcp [ 683.292647] LNet: Removed LNI 10.1.2.3@tcp [ 683.321539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 683.388108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 683.411672] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 683.415454] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 683.760754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 683.781447] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 684.589728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 684.607664] LNet: 43084:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 684.611420] LNet: 43084:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 684.628968] LNetError: 43084:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 684.639365] LNetError: 43084:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 684.657375] LNet: Removed LNI 192.168.203.32@tcp [ 684.661323] LNet: Removed LNI 10.1.2.3@tcp [ 684.681504] LNet: Removed LNI 192.168.203.32@tcp2 [ 684.707927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 684.777146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 684.792774] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 684.800143] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 684.803288] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 690.065184] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 18:09:16 (1763334556) [ 690.238699] LNet: 43681:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 690.244498] LNetError: 43681:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 690.249401] LNetError: 43681:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 690.254658] LNet: Removed LNI 192.168.203.32@tcp [ 690.259672] LNet: Removed LNI 10.1.2.3@tcp [ 690.269499] LNet: Removed LNI 192.168.203.32@tcp2 [ 690.799192] Key type .llcrypt unregistered [ 690.800656] Key type ._llcrypt unregistered [ 695.972675] Key type ._llcrypt registered [ 695.977284] Key type .llcrypt registered [ 696.243349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 696.272055] alg: No test for adler32 (adler32-zlib) [ 697.272123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.322167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 697.347731] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 698.065766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.084753] LNet: 44072:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 698.089981] LNetError: 44072:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 698.113487] LNet: Removed LNI 192.168.203.32@tcp [ 698.126584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.201723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 698.225309] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 698.572555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.585855] LNet: 44102:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 698.591439] LNetError: 44102:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 698.604233] LNet: Removed LNI 192.168.203.32@tcp [ 698.635205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.697730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 698.715473] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 699.000359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 699.032529] LNet: Removed LNI 192.168.203.32@tcp [ 699.045847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.096934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 699.123328] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 699.475947] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 699.563343] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 699.569704] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 699.573883] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 701.816925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 701.834835] LNet: 44210:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 701.838989] LNet: 44210:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 701.843226] LNetError: 44210:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 701.846448] LNetError: 44210:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 701.859409] LNet: Removed LNI 192.168.203.32@tcp [ 701.874250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 701.907453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 701.924410] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 706.729778] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 18:09:33 (1763334573) [ 706.897714] LNet: 44803:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 706.902767] LNetError: 44803:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 706.936667] LNet: Removed LNI 192.168.203.32@tcp [ 707.619664] Key type .llcrypt unregistered [ 707.623515] Key type ._llcrypt unregistered [ 711.537679] Key type ._llcrypt registered [ 711.539445] Key type .llcrypt registered [ 711.718208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 711.743469] alg: No test for adler32 (adler32-zlib) [ 712.684577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.707953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 712.720931] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 713.422785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.439122] LNet: 45200:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 713.460763] LNetError: 45200:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 713.476918] LNet: Removed LNI 192.168.203.32@tcp [ 713.495920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.591398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 713.633232] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 718.366510] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 18:09:45 (1763334585) [ 718.581938] LNet: 45792:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 718.585527] LNetError: 45792:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 718.598424] LNet: Removed LNI 192.168.203.32@tcp [ 718.941478] Key type .llcrypt unregistered [ 718.943393] Key type ._llcrypt unregistered [ 722.506152] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 722.621682] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 722.626050] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 722.632683] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 725.039591] Key type ._llcrypt registered [ 725.043828] Key type .llcrypt registered [ 725.227623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 725.237176] alg: No test for adler32 (adler32-zlib) [ 726.081034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 726.122799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 726.145668] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 726.697346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 726.711461] LNet: 46235:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 726.715495] LNetError: 46235:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 726.728689] LNet: Removed LNI 192.168.203.32@tcp [ 726.743226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 726.789114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 726.804946] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 731.157173] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 18:09:58 (1763334598) [ 731.214930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 731.236886] LNet: 46826:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 731.241418] LNetError: 46826:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 731.252637] LNet: Removed LNI 192.168.203.32@tcp [ 731.275760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 731.358351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 735.467975] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 18:10:02 (1763334602) [ 735.564502] LNet: 47414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 735.979468] Key type .llcrypt unregistered [ 735.983427] Key type ._llcrypt unregistered [ 739.922518] Key type ._llcrypt registered [ 739.924019] Key type .llcrypt registered [ 740.058846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 740.067570] alg: No test for adler32 (adler32-zlib) [ 740.986101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 741.030300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 741.054897] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 741.693929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 741.712605] LNet: 47808:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 741.716699] LNetError: 47808:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 741.730048] LNet: Removed LNI 192.168.203.32@tcp [ 741.743949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 741.791900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 741.812726] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 746.743908] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 18:10:13 (1763334613) [ 746.860207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 746.901903] LNet: 48395:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 746.920161] LNetError: 48395:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 746.930973] LNet: Removed LNI 192.168.203.32@tcp [ 746.986920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 747.044201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 752.052118] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 18:10:18 (1763334618) [ 753.228507] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 754.641738] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 18:10:21 (1763334621) [ 755.869113] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 757.202984] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 18:10:23 (1763334623) [ 758.454512] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 759.901640] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 18:10:26 (1763334626) [ 761.134652] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 762.568470] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 18:10:29 (1763334629) [ 763.838559] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 765.249543] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 18:10:32 (1763334632) [ 765.537583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 765.556149] LNet: 50452:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 765.582305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 765.636239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 765.654971] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 766.290915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.32 [ 766.307773] LNet: Removed LNI 192.168.203.32@tcp [ 771.525590] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 18:10:38 (1763334638) [ 771.805905] LNet: 51156:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 771.811971] LNetError: 51156:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 771.822885] LNet: Removed LNI 192.168.203.32@tcp [ 772.323622] Key type .llcrypt unregistered [ 772.325364] Key type ._llcrypt unregistered [ 777.052932] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 777.221394] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 777.223778] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 777.232429] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 779.749988] Key type ._llcrypt registered [ 779.751607] Key type .llcrypt registered [ 779.883366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 779.895354] alg: No test for adler32 (adler32-zlib) [ 781.210137] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 786.468464] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 18:10:53 (1763334653) [ 786.799632] LNet: 52056:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 786.810406] LNetError: 52056:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 786.823059] LNet: Removed LNI 10.1.2.3@tcp [ 787.296160] Key type .llcrypt unregistered [ 787.297985] Key type ._llcrypt unregistered [ 791.889545] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 792.069631] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 792.072745] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 792.077429] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 794.751927] Key type ._llcrypt registered [ 794.753661] Key type .llcrypt registered [ 794.930660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 794.951694] alg: No test for adler32 (adler32-zlib) [ 795.858553] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 801.942436] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 18:11:08 (1763334668) [ 802.227720] LNet: 52945:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 802.244368] LNetError: 52945:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 802.263813] LNet: Removed LNI 10.1.2.3@tcp [ 802.859431] Key type .llcrypt unregistered [ 802.861318] Key type ._llcrypt unregistered [ 808.709146] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 808.883169] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 808.891876] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 808.902358] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 811.492952] Key type ._llcrypt registered [ 811.494700] Key type .llcrypt registered [ 811.612501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 811.623196] alg: No test for adler32 (adler32-zlib) [ 812.528742] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 818.270544] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 18:11:24 (1763334684) [ 818.371992] LNet: 53825:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 818.386725] LNetError: 53825:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 818.403760] LNet: Removed LNI 10.1.2.3@tcp [ 819.188478] Key type .llcrypt unregistered [ 819.193434] Key type ._llcrypt unregistered [ 824.380214] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 824.588410] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 824.596810] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 824.601556] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 827.102663] Key type ._llcrypt registered [ 827.104468] Key type .llcrypt registered [ 827.287953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 827.301391] alg: No test for adler32 (adler32-zlib) [ 828.227101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 828.313541] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 835.587994] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 18:11:42 (1763334702) [ 836.062994] LNet: 55059:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 836.073623] LNetError: 55059:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 836.086663] LNet: Removed LNI 10.1.2.3@tcp [ 836.593459] Key type .llcrypt unregistered [ 836.602074] Key type ._llcrypt unregistered [ 841.816652] Key type ._llcrypt registered [ 841.818373] Key type .llcrypt registered [ 842.073750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 842.093018] alg: No test for adler32 (adler32-zlib) [ 843.036634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 843.082951] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 849.055717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.132@tcp [ 849.599889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_interrupt -r 1 [ 849.644606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_interrupt -r 1 [ 849.683712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 849.703581] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 849.875809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 849.901804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 849.931766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 849.966895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 849.996877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 850.016749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 850.038146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 850.066850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 850.087124] LNet: 55493:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 850.097709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 850.127695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 850.157900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 850.184866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 850.214353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 850.250406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 850.276045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 850.724308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_dropped -r 1 [ 850.749407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_dropped -r 1 [ 850.774976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 850.795479] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 850.961810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 850.987868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 851.016870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 851.039776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 851.070339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 851.097293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 851.125209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 851.159449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 851.173176] LNet: 55602:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 851.204188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 851.233427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 851.263891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 851.283249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 851.328334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 851.356712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 851.374152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 851.789267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_aborted -r 1 [ 851.823118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_aborted -r 1 [ 851.871710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 851.901149] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 852.101843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 852.135968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 852.176390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 852.205186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 852.238590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 852.265280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 852.321834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 852.354704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 852.367381] LNet: 55712:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 852.383276] LNet: 1 local NIs in recovery (showing 1): 192.168.203.32@tcp [ 852.392539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 852.415238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 852.446598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 852.477435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 852.503576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 852.537698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 852.565546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 852.944532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_no_route -r 1 [ 852.982132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_no_route -r 1 [ 853.012765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 853.190090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 853.232890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 853.255083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 853.272923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 853.309507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 853.337829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 853.369598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 853.401452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 853.428255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 853.448158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 853.465413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 853.499876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 853.524052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 853.551441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 853.584833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 853.993291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_timeout -r 1 [ 854.025828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_timeout -r 1 [ 854.056556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 854.078780] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 854.082641] LNet: Skipped 2 previous similar messages [ 854.277097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 854.301257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 854.342601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 854.367476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 854.394928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 854.424696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 854.454314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 854.493535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 854.513853] LNet: 55930:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 854.518372] LNet: 55930:0:(api-ni.c:340:recovery_interval_set()) Skipped 1 previous similar message [ 854.533733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 854.579317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 854.623686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 854.662366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 854.689651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 854.713240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 854.748972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 855.127802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_error -r 1 [ 855.161360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_error -r 1 [ 855.188227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 855.441692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 855.489669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 855.511603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 855.529682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 855.585312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 855.631603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 855.664103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 855.690621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 855.727459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 855.760172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 855.782392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 855.814144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 855.856338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 855.887179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 855.916957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 856.137263] LNet: 56079:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 856.142273] LNetError: 56079:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 856.160275] LNet: Removed LNI 192.168.203.32@tcp [ 856.508572] Key type .llcrypt unregistered [ 856.513269] Key type ._llcrypt unregistered [ 871.838627] Key type ._llcrypt registered [ 871.840264] Key type .llcrypt registered [ 872.918661] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 18:12:19 (1763334739) [ 873.490455] Key type .llcrypt unregistered [ 873.494271] Key type ._llcrypt unregistered [ 877.697439] Key type ._llcrypt registered [ 877.699055] Key type .llcrypt registered [ 877.939953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 877.959213] alg: No test for adler32 (adler32-zlib) [ 878.854116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 878.876321] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 885.423309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.132@tcp [ 894.229252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 894.269440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 894.304237] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 895.840215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_interrupt -r 1 [ 895.899630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e local_interrupt -r 1 [ 895.963353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_interrupt -r 1 [ 896.037332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e local_interrupt -r 1 [ 896.061074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e local_interrupt -r 1 [ 896.093473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e local_interrupt -r 1 [ 896.137488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e local_interrupt -r 1 [ 896.188441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e local_interrupt -r 1 [ 896.232656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 896.261819] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 896.266367] LNet: 1 local NIs in recovery (showing 1): 192.168.203.32@tcp [ 896.545590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 896.582513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 896.617809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 896.653022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 896.696335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 896.734476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 896.805046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 896.883738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 896.937706] LNet: 57540:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 896.959297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 897.011135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 897.056602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 897.086756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 897.151984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 897.174577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 897.207250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 897.861042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_dropped -r 1 [ 897.907562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e local_dropped -r 1 [ 897.930654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_dropped -r 1 [ 897.952586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e local_dropped -r 1 [ 897.992141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e local_dropped -r 1 [ 898.031347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e local_dropped -r 1 [ 898.070762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e local_dropped -r 1 [ 898.096053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e local_dropped -r 1 [ 898.138348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 898.167172] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 898.171121] LNet: Skipped 2 previous similar messages [ 898.451670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 898.481620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 898.525413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 898.553472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 898.591390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 898.621769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 898.667323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 898.704310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 898.751768] LNet: 57661:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 898.794129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 898.831656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 898.881816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 898.922838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 898.968814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 899.025560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 899.078482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 899.732710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_aborted -r 1 [ 899.775887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e local_aborted -r 1 [ 899.806247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_aborted -r 1 [ 899.865149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e local_aborted -r 1 [ 899.895657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e local_aborted -r 1 [ 899.957247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e local_aborted -r 1 [ 899.985130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e local_aborted -r 1 [ 900.017273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e local_aborted -r 1 [ 900.051738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 900.074191] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 900.078434] LNet: Skipped 2 previous similar messages [ 900.238534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 900.262438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 900.283593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 900.316974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 900.346148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 900.378251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 900.406806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 900.433728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 900.450436] LNet: 57783:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 900.466373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 900.492277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 900.527211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 900.552498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 900.577270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 900.638066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 900.682640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 901.099141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_no_route -r 1 [ 901.126362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e local_no_route -r 1 [ 901.166637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_no_route -r 1 [ 901.222731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e local_no_route -r 1 [ 901.255394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e local_no_route -r 1 [ 901.277500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e local_no_route -r 1 [ 901.329200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e local_no_route -r 1 [ 901.380314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e local_no_route -r 1 [ 901.412437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 901.615608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 901.641986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 901.671410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 901.705416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 901.734634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 901.760595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 901.789577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 901.817293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 901.848140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 901.864913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 901.884419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 901.915310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 901.950568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 901.977425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 901.999123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 902.460387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_timeout -r 1 [ 902.488757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e local_timeout -r 1 [ 902.512218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_timeout -r 1 [ 902.543938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e local_timeout -r 1 [ 902.565797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e local_timeout -r 1 [ 902.611963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e local_timeout -r 1 [ 902.642689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e local_timeout -r 1 [ 902.672282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e local_timeout -r 1 [ 902.699183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 902.714925] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 902.719065] LNet: Skipped 5 previous similar messages [ 902.871296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 902.897263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 902.923630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 902.949730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 902.980104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 903.016888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 903.057439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 903.089652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 903.109474] LNet: 58025:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 903.124319] LNet: 58025:0:(api-ni.c:340:recovery_interval_set()) Skipped 1 previous similar message [ 903.143739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 903.172760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 903.197594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 903.221926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 903.244770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 903.269612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 903.288666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 903.712743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_error -r 1 [ 903.736860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e local_error -r 1 [ 903.762838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_error -r 1 [ 903.789478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e local_error -r 1 [ 903.819983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e local_error -r 1 [ 903.848321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e local_error -r 1 [ 903.878214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e local_error -r 1 [ 903.901981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e local_error -r 1 [ 903.934660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 904.126862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 904.155022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 904.195388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 904.227168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 904.253879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 904.294538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 904.340268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 904.371525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 904.401379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 904.451904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 904.493797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 904.525921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 904.555852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 904.587214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 904.628295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 909.417715] LNet: 58222:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.431473] LNetError: 58222:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 909.441854] LNet: Removed LNI 192.168.203.32@tcp [ 909.450494] LNet: Removed LNI 192.168.203.32@tcp1 [ 909.935966] Key type .llcrypt unregistered [ 909.938461] Key type ._llcrypt unregistered [ 923.140478] Key type ._llcrypt registered [ 923.150711] Key type .llcrypt registered [ 924.026982] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 18:13:10 (1763334790) [ 924.801488] Key type .llcrypt unregistered [ 924.803400] Key type ._llcrypt unregistered [ 929.231158] Key type ._llcrypt registered [ 929.233272] Key type .llcrypt registered [ 929.468633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 929.488667] alg: No test for adler32 (adler32-zlib) [ 930.371335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 930.400344] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 936.781615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.132@tcp [ 937.441843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e remote_dropped -r 1 [ 937.487980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e remote_dropped -r 1 [ 937.536435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 937.566798] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 937.786941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 937.825413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 937.854646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 937.883933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 937.921653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 937.948710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 937.978982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 938.007239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 938.029682] LNet: 59466:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 938.046674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 938.067679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 938.089611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 938.120803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 938.146234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 938.170315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 938.197227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 938.664860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e remote_error -r 1 [ 938.708275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e remote_error -r 1 [ 938.747520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 938.779746] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 939.012983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 939.042941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 939.069927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 939.099990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 939.151502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 939.180562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 939.223227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 939.248113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 939.284996] LNet: 59575:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 939.312276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 939.346238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 939.378213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 939.421173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 939.456438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 939.487916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 939.511201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 939.957687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e remote_timeout -r 1 [ 940.043983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e remote_timeout -r 1 [ 940.097553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 940.126041] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 940.410645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 940.432223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 940.462797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 940.487687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 940.525245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 940.570790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 940.601319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 940.636240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 940.660211] LNet: 59686:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 940.676613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 940.729735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 940.772608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 940.826950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 940.859855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 940.890990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 940.923095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 941.203990] LNet: 59726:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 941.213581] LNetError: 59726:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 941.239335] LNet: Removed LNI 192.168.203.32@tcp [ 941.658622] Key type .llcrypt unregistered [ 941.660553] Key type ._llcrypt unregistered [ 956.857697] Key type ._llcrypt registered [ 956.859418] Key type .llcrypt registered [ 957.972306] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 18:13:44 (1763334824) [ 958.989324] Key type .llcrypt unregistered [ 958.992942] Key type ._llcrypt unregistered [ 964.008467] Key type ._llcrypt registered [ 964.010358] Key type .llcrypt registered [ 964.222627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 964.253855] alg: No test for adler32 (adler32-zlib) [ 965.225887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 965.270139] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 971.740859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.132@tcp [ 978.478521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 978.533617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 978.566452] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 979.824358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e remote_dropped -r 1 [ 979.850173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e remote_dropped -r 1 [ 979.873767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e remote_dropped -r 1 [ 979.901218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e remote_dropped -r 1 [ 979.922228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e remote_dropped -r 1 [ 979.957937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e remote_dropped -r 1 [ 979.988561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e remote_dropped -r 1 [ 980.019947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e remote_dropped -r 1 [ 980.077702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 980.113037] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 980.120252] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.132@tcp [ 980.380502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 980.410406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 980.446603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 980.478859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 980.501791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 980.535096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 980.585099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 980.617864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 980.638461] LNet: 61186:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 980.658801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 980.695389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 980.748173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 980.800140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 980.825670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 980.858989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 980.895248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 980.963786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 981.498415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e remote_error -r 1 [ 981.535742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e remote_error -r 1 [ 981.576478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e remote_error -r 1 [ 981.607745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e remote_error -r 1 [ 981.653586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e remote_error -r 1 [ 981.685625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e remote_error -r 1 [ 981.715317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e remote_error -r 1 [ 981.752607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e remote_error -r 1 [ 981.781698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 981.816786] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 981.822226] LNet: Skipped 2 previous similar messages [ 982.030911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 982.060454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 982.088539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 982.115871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 982.146938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 982.180511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 982.203597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 982.235394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 982.255223] LNet: 61310:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 982.269617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 982.293932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 982.320766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 982.347337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 982.373329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 982.402945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 982.428899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 982.466152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 982.885234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e remote_timeout -r 1 [ 982.921621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e remote_timeout -r 1 [ 982.952673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e remote_timeout -r 1 [ 982.975189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e remote_timeout -r 1 [ 983.016360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e remote_timeout -r 1 [ 983.055590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e remote_timeout -r 1 [ 983.100299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e remote_timeout -r 1 [ 983.132404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e remote_timeout -r 1 [ 983.170739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 983.195897] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 983.200159] LNet: Skipped 1 previous similar message [ 983.343658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 983.371211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 983.393557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 983.413928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 983.437612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 983.466505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 983.503527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 983.545796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 983.570068] LNet: 61433:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 983.595479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 983.627150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 983.652736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 983.685529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 983.728438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 983.769560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 983.801700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 983.837162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 987.845410] LNet: 61510:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 987.857427] LNetError: 61510:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 987.866506] LNet: Removed LNI 192.168.203.32@tcp [ 987.888420] LNet: Removed LNI 192.168.203.32@tcp1 [ 988.378757] Key type .llcrypt unregistered [ 988.383373] Key type ._llcrypt unregistered [ 1002.995255] Key type ._llcrypt registered [ 1002.997092] Key type .llcrypt registered [ 1004.174049] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 18:14:30 (1763334870) [ 1004.584530] Key type .llcrypt unregistered [ 1004.586376] Key type ._llcrypt unregistered [ 1009.565359] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1009.775281] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1009.777169] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1009.817633] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1012.561688] Key type ._llcrypt registered [ 1012.564393] Key type .llcrypt registered [ 1012.698744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1012.721803] alg: No test for adler32 (adler32-zlib) [ 1013.641421] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1013.693906] LNet: 62597:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1013.714674] LNetError: 62597:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1013.740627] LNet: Removed LNI 192.168.203.32@tcp [ 1014.383421] Key type .llcrypt unregistered [ 1014.389386] Key type ._llcrypt unregistered [ 1020.012861] Key type ._llcrypt registered [ 1020.020654] Key type .llcrypt registered [ 1020.168464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1020.191439] alg: No test for adler32 (adler32-zlib) [ 1021.077666] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1021.081596] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 1021.107850] LNet: 62866:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1021.114498] LNetError: 62866:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1021.119087] LNet: Removed LNI 192.168.203.32@tcp [ 1021.134369] LNet: Removed LNI 10.1.2.3@tcp1 [ 1021.708267] Key type .llcrypt unregistered [ 1021.712311] Key type ._llcrypt unregistered [ 1026.544332] Key type ._llcrypt registered [ 1026.546431] Key type .llcrypt registered [ 1026.683346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1026.702764] alg: No test for adler32 (adler32-zlib) [ 1027.603901] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1027.606640] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1027.632720] LNet: 63135:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1027.639311] LNetError: 63135:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1027.643174] LNet: Removed LNI 192.168.203.32@tcp [ 1027.653798] LNet: Removed LNI 10.1.2.3@tcp [ 1028.311247] Key type .llcrypt unregistered [ 1028.313742] Key type ._llcrypt unregistered [ 1033.217654] Key type ._llcrypt registered [ 1033.219327] Key type .llcrypt registered [ 1033.481360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1033.501406] alg: No test for adler32 (adler32-zlib) [ 1034.552626] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1034.554854] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1034.614884] LNet: 63405:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1034.622244] LNetError: 63405:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1034.627177] LNet: Removed LNI 192.168.203.32@tcp [ 1034.636677] LNet: Removed LNI 10.1.2.3@tcp [ 1035.324584] Key type .llcrypt unregistered [ 1035.326295] Key type ._llcrypt unregistered [ 1041.647841] Key type ._llcrypt registered [ 1041.654791] Key type .llcrypt registered [ 1041.823471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1041.841043] alg: No test for adler32 (adler32-zlib) [ 1042.734971] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1042.739338] LNetError: 63666:0:(config.c:1557:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 1042.746268] LNetError: Error -100 starting up LNI tcp [ 1042.757655] LNet: Removed LNI 192.168.203.32@tcp [ 1043.365774] Key type .llcrypt unregistered [ 1043.368663] Key type ._llcrypt unregistered [ 1053.676420] Key type ._llcrypt registered [ 1053.677890] Key type .llcrypt registered [ 1055.063021] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 18:15:21 (1763334921) [ 1055.695409] Key type .llcrypt unregistered [ 1055.698797] Key type ._llcrypt unregistered [ 1061.435184] Key type ._llcrypt registered [ 1061.436862] Key type .llcrypt registered [ 1061.639853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1061.655764] alg: No test for adler32 (adler32-zlib) [ 1062.610914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1062.656880] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1069.708713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.132@tcp [ 1070.395894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e network_timeout -r 1 [ 1070.417816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e network_timeout -r 1 [ 1070.454020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.132@tcp [ 1070.469476] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1070.688453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1070.721738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1070.771171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1070.809376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1070.858622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1070.899129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1070.909344] LNet: 1 local NIs in recovery (showing 1): 192.168.203.32@tcp [ 1070.962653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1071.003873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1071.026707] LNet: 64916:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1071.040610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1071.080352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1071.113170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1071.172377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1071.214741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1071.255311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1071.480538] LNet: 64954:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1071.488470] LNetError: 64954:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1071.504322] LNet: Removed LNI 192.168.203.32@tcp [ 1071.932656] Key type .llcrypt unregistered [ 1071.935060] Key type ._llcrypt unregistered [ 1086.442112] Key type ._llcrypt registered [ 1086.446726] Key type .llcrypt registered [ 1086.657154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1086.664266] alg: No test for adler32 (adler32-zlib) [ 1087.547223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1087.568588] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1093.180574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.132@tcp [ 1099.268516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1099.292316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1099.306776] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 1100.173487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e network_timeout -r 1 [ 1100.206468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e network_timeout -r 1 [ 1100.234330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e network_timeout -r 1 [ 1100.258339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e network_timeout -r 1 [ 1100.285841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e network_timeout -r 1 [ 1100.318925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e network_timeout -r 1 [ 1100.343336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e network_timeout -r 1 [ 1100.374559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e network_timeout -r 1 [ 1100.404203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.132@tcp [ 1100.417557] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1100.574676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1100.594573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1100.616527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1100.639615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1100.660466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1100.685847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1100.705574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1100.736873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1100.753324] LNet: 65827:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1100.765878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1100.792942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1100.810820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1100.829123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1100.848905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1100.878662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1104.452545] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1104.794162] LNet: 65900:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1104.800535] LNetError: 65900:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1104.806503] LNet: Removed LNI 192.168.203.32@tcp [ 1104.814808] LNet: Removed LNI 192.168.203.32@tcp1 [ 1105.153544] Key type .llcrypt unregistered [ 1105.157305] Key type ._llcrypt unregistered [ 1116.233969] Key type ._llcrypt registered [ 1116.236395] Key type .llcrypt registered [ 1117.369820] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 18:16:23 (1763334983) [ 1117.613522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1117.625053] alg: No test for adler32 (adler32-zlib) [ 1118.506736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1118.557175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1118.579973] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1119.146625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1119.169163] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 1119.891509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 1119.947856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1119.987670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1120.013650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1120.043855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1120.069233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1120.107983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.32@tcp [ 1120.126243] LNetError: 66997:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.203.32@tcp to recovery [ 1120.609284] LNet: 1 local NIs in recovery (showing 1): 192.168.203.32@tcp [ 1120.610237] LNet: There was an unexpected network error while writing to 192.168.203.32: rc = -22 [ 1122.655463] LNet: There was an unexpected network error while writing to 192.168.203.32: rc = -22 [ 1123.506274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1126.760241] LNet: There was an unexpected network error while writing to 192.168.203.32: rc = -22 [ 1126.930689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1126.961592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1127.064421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1127.092781] LNet: 67104:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1127.097954] LNetError: 67104:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1127.103897] LNet: Removed LNI 192.168.203.32@tcp [ 1127.117794] LNet: Removed LNI 192.168.203.32@tcp1 [ 1127.162604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1127.213695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1127.240531] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1127.824373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1127.836254] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 1128.594571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 1128.631460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1128.697933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1128.735672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1128.763869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1128.797262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1128.822575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1128.851638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.32@tcp [ 1128.863505] LNetError: 67376:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.203.32@tcp to recovery [ 1129.250972] LNet: There was an unexpected network error while writing to 192.168.203.32: rc = -22 [ 1132.414723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1135.394746] LNet: There was an unexpected network error while writing to 192.168.203.32: rc = -22 [ 1135.399141] LNet: Skipped 1 previous similar message [ 1140.148082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1140.181891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1140.203526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1146.318954] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 18:16:52 (1763335012) [ 1146.379120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1146.398501] LNet: 68086:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1146.404177] LNetError: 68086:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1146.407773] LNetError: 68086:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1146.411726] LNet: Removed LNI 192.168.203.32@tcp [ 1146.423258] LNet: Removed LNI 192.168.203.32@tcp1 [ 1146.447527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1146.534522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1146.562859] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1147.162822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1147.179108] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 1147.880349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 1147.947946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1147.980549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1148.014577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1148.050938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1148.080759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1148.109551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.32@tcp --health 0 [ 1148.206621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1148.511270] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.32@tcp [ 1149.537414] LNet: There was an unexpected network error while writing to 192.168.203.32: rc = -22 [ 1149.541620] LNet: Skipped 1 previous similar message [ 1164.369501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1164.498363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1164.532845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1164.560985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.32@tcp --health 500 [ 1164.645732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1171.176409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1171.252121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1171.267438] LNet: 68596:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1171.275937] LNetError: 68596:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1171.280831] LNetError: 68596:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1171.284570] LNet: Removed LNI 192.168.203.32@tcp [ 1171.301614] LNet: Removed LNI 192.168.203.32@tcp1 [ 1171.379647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1171.435218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1171.470243] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1172.029192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1172.049289] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 1172.844229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 1172.879404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1172.942689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1172.967318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1172.996822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1173.023399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1173.050673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1173.076877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.32@tcp --health 0 [ 1174.495819] LNet: There was an unexpected network error while writing to 192.168.203.32: rc = -22 [ 1174.508157] LNet: Skipped 3 previous similar messages [ 1175.514928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1185.741938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1185.788422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1185.815852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1191.952051] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 18:17:38 (1763335058) [ 1192.295816] LNet: 69653:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1192.304573] LNetError: 69653:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1192.311212] LNetError: 69653:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1192.317066] LNet: Removed LNI 192.168.203.32@tcp [ 1192.329675] LNet: Removed LNI 192.168.203.32@tcp1 [ 1192.808254] Key type .llcrypt unregistered [ 1192.810062] Key type ._llcrypt unregistered [ 1197.786704] Key type ._llcrypt registered [ 1197.788307] Key type .llcrypt registered [ 1198.025448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1198.049123] alg: No test for adler32 (adler32-zlib) [ 1199.104165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1199.176558] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1207.743262] Lustre: DEBUG MARKER: Initial discovery [ 1207.808619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.132@tcp [ 1210.093168] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1210.293960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1210.353836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.132@tcp [ 1210.381302] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1210.399303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1210.426134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.132@tcp -r 1 -m GET -l 3 [ 1210.470596] LNet: 1 local NIs in recovery (showing 1): 192.168.203.32@tcp [ 1210.479661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.132@tcp -r 1 -m GET -e local_error [ 1210.502186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.132@tcp -r 1 -m PUT -l 6 [ 1211.735793] Lustre: DEBUG MARKER: Force oleg332-client.virtnet to discover 192.168.203.132@tcp (in background) [ 1211.905537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.132@tcp [ 1213.012247] Lustre: DEBUG MARKER: Force oleg332-server to discover 192.168.203.32@tcp [ 1215.008592] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1215.886667] Lustre: DEBUG MARKER: Wait for 70411 [ 1217.279552] Lustre: DEBUG MARKER: Finished wait on 70411 [ 1217.333131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1217.368863] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1217.398475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1217.640541] LNet: 70859:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1217.644319] LNetError: 70859:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1218.661843] LNet: Removed LNI 192.168.203.32@tcp [ 1219.032583] Key type .llcrypt unregistered [ 1219.034260] Key type ._llcrypt unregistered [ 1236.602363] Key type ._llcrypt registered [ 1236.611851] Key type .llcrypt registered [ 1238.122948] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 18:18:24 (1763335104) [ 1238.625950] Key type .llcrypt unregistered [ 1238.627791] Key type ._llcrypt unregistered [ 1243.426733] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1243.544426] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1243.549289] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1243.554526] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1246.097797] Key type ._llcrypt registered [ 1246.099484] Key type .llcrypt registered [ 1246.378198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1246.396800] alg: No test for adler32 (adler32-zlib) [ 1247.279439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1247.315719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1247.336681] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1248.213708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1248.258716] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1258.092995] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 18:18:44 (1763335124) [ 1258.206802] LNet: 72916:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1258.211867] LNetError: 72916:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1258.231785] LNet: Removed LNI 192.168.203.32@tcp [ 1258.264220] LNet: Removed LNI 10.1.2.3@tcp [ 1258.946787] Key type .llcrypt unregistered [ 1258.948418] Key type ._llcrypt unregistered [ 1264.056060] Key type ._llcrypt registered [ 1264.057523] Key type .llcrypt registered [ 1264.286535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1264.294375] alg: No test for adler32 (adler32-zlib) [ 1265.197619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1265.262423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1265.292714] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1266.081891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1266.098502] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1267.023621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.32@tcp [ 1268.063686] LNetError: Unexpected error -22 connecting to 192.168.203.32@tcp at host 192.168.203.32:988 [ 1272.697036] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 18:18:59 (1763335139) [ 1272.834606] LNet: 74046:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1272.840978] LNetError: 74046:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1272.849707] LNet: Removed LNI 192.168.203.32@tcp [ 1272.861947] LNet: Removed LNI 10.1.2.3@tcp [ 1273.460553] Key type .llcrypt unregistered [ 1273.464483] Key type ._llcrypt unregistered [ 1278.632140] Key type ._llcrypt registered [ 1278.633783] Key type .llcrypt registered [ 1278.955342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1278.984348] alg: No test for adler32 (adler32-zlib) [ 1279.913730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1279.967317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1279.984682] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 1280.711869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1280.738777] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 1281.698668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.32@tcp1 --nid 192.168.203.32@tcp2 [ 1285.889351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.32@tcp1 192.168.203.32@tcp2 [ 1285.922485] LNetError: 74882:0:(lib-move.c:2264:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.32@tcp1 to 192.168.203.32@tcp2 - Route table may be misconfigured [ 1285.937936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.32@tcp2 192.168.203.32@tcp1 [ 1291.828205] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 18:19:18 (1763335158) [ 1292.007753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1292.027516] LNet: 75454:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1292.037331] LNetError: 75454:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1292.044265] LNet: Removed LNI 192.168.203.32@tcp1 [ 1293.097386] LNet: Removed LNI 192.168.203.32@tcp2 [ 1293.156370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1293.235356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1293.253117] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1294.069236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1294.091247] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 1294.894672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 1295.007461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.32@tcp 192.168.203.32@tcp [ 1295.050963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.32@tcp1 192.168.203.32@tcp1 [ 1295.086869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.32@tcp -d 192.168.203.32@tcp -e network_timeout [ 1295.116023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -e network_timeout [ 1295.177403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -e network_timeout [ 1295.217378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -e network_timeout [ 1295.255336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1295.283481] LNet: There was an unexpected network error while writing to 192.168.203.32: rc = -22 [ 1295.316432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1295.431077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1295.555176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1296.287270] LNet: 1 local NIs in recovery (showing 1): 192.168.203.32@tcp [ 1302.076894] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 18:19:28 (1763335168) [ 1302.194493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1302.250138] LNet: 76311:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1302.272919] LNetError: 76311:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1302.275707] LNetError: 76311:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1302.288988] LNet: Removed LNI 192.168.203.32@tcp [ 1303.337154] LNet: Removed LNI 192.168.203.32@tcp1 [ 1303.371758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1303.458423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1303.710421] LNet: 76352:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1304.065124] Key type .llcrypt unregistered [ 1304.069327] Key type ._llcrypt unregistered [ 1314.828158] Key type ._llcrypt registered [ 1314.829723] Key type .llcrypt registered [ 1316.155441] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 18:19:42 (1763335182) [ 1316.509833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1316.534261] alg: No test for adler32 (adler32-zlib) [ 1317.479519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1318.825681] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1320.724949] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 18:19:47 (1763335187) [ 1320.986795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1321.003251] LNet: 77447:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1321.058890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1321.123255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1321.142794] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1321.789507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1321.814321] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 1322.796658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1322.836275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp1 [ 1322.867185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp1 [ 1329.664394] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 18:19:56 (1763335196) [ 1331.663720] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg332-server [ 1333.381607] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 18:19:59 (1763335199) [ 1335.183954] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg332-server [ 1337.460610] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 18:20:03 (1763335203) [ 1339.411709] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg332-server [ 1341.193949] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 18:20:07 (1763335207) [ 1342.967819] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg332-server [ 1344.686599] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 18:20:11 (1763335211) [ 1346.523544] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg332-server [ 1348.615084] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 18:20:14 (1763335214) [ 1351.264911] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg332-server [ 1353.205775] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 18:20:19 (1763335219) [ 1355.034645] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg332-server [ 1356.807738] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 18:20:23 (1763335223) [ 1358.607471] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg332-server [ 1360.669109] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 18:20:26 (1763335226) [ 1362.280553] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg332-server [ 1364.731803] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 18:20:30 (1763335230) [ 1364.850776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1364.882310] LNet: 81320:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1364.888378] LNetError: 81320:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1364.916846] LNet: Removed LNI 192.168.203.32@tcp [ 1365.925933] LNet: Removed LNI 192.168.203.32@tcp1 [ 1365.963586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1366.041136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1366.060773] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1366.880513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1366.990438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1367.262769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1367.292735] LNet: 81473:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1367.308155] LNetError: 81473:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1367.319872] LNetError: 81473:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1367.341952] LNet: Removed LNI 192.168.203.32@tcp [ 1367.361531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1367.418516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1367.456493] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1368.220912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1368.429620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1368.469911] LNetError: 81618:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1368.482258] LNet: Removed LNI 192.168.203.32@tcp [ 1368.506412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1368.562703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1368.602691] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1369.417166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1369.480507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1369.669314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1369.687927] LNet: 81766:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1369.692450] LNet: 81766:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1370.731606] LNet: Removed LNI 192.168.203.32@tcp [ 1370.761535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1370.809397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1370.845526] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1371.464712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1371.714432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1371.756719] LNetError: 81911:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1371.761539] LNetError: 81911:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1371.777905] LNet: Removed LNI 192.168.203.32@tcp [ 1371.815996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1371.887222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1371.924944] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1372.585855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1372.659736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1372.935453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1373.995593] LNet: Removed LNI 192.168.203.32@tcp [ 1374.033947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1374.108894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1374.137950] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1374.928637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1375.224359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1375.244868] LNet: 82203:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1375.248516] LNet: 82203:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1376.293786] LNet: Removed LNI 192.168.203.32@tcp [ 1376.315273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1376.377833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1376.403317] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1377.058895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1377.143502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1377.461914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1377.494454] LNetError: 82351:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1377.497876] LNetError: 82351:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1377.524111] LNet: Removed LNI 192.168.203.32@tcp [ 1377.559559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1377.636148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1377.692346] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1378.567625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1378.847693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1379.944134] LNet: Removed LNI 192.168.203.32@tcp [ 1379.972973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1380.034786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1380.110129] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1381.278527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1381.379290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1382.922665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1383.987988] LNet: Removed LNI 192.168.203.32@tcp [ 1384.013538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1384.057392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1384.081935] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1384.699136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1385.038569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1385.066297] LNet: 82798:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1385.069616] LNet: 82798:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1386.155657] LNet: Removed LNI 192.168.203.32@tcp [ 1386.198604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1386.278855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1386.331420] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1387.127460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1387.175577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1387.479406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1387.517400] LNetError: 82945:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1387.521118] LNetError: 82945:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1387.544148] LNet: Removed LNI 192.168.203.32@tcp [ 1387.571974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1387.656702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1387.690845] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1388.416850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1388.752391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1389.797773] LNet: Removed LNI 192.168.203.32@tcp [ 1389.820205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1389.885693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1389.920337] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1390.531388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1390.581843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1390.603206] LNet: Removed LNI 192.168.203.32@tcp [ 1390.614581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1390.689135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1390.727926] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1391.513950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1397.884829] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 18:21:04 (1763335264) [ 1397.967198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1398.017419] LNet: Removed LNI 192.168.203.32@tcp [ 1398.060134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1398.122722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1398.144514] LNet: Added LNI 192.168.203.32@tcp231 [8/256/0/180] [ 1399.074815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1399.116825] LNet: Removed LNI 192.168.203.32@tcp231 [ 1399.135296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1399.205480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1399.228993] LNet: Added LNI 192.168.203.32@tcp231 [8/256/0/180] [ 1399.234913] LNet: 84073:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1399.555363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1399.601072] LNet: Removed LNI 192.168.203.32@tcp231 [ 1399.648445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1399.683360] LNet: Added LNI 192.168.203.32@tcp231 [8/256/0/180] [ 1400.644100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1400.668474] LNet: Removed LNI 192.168.203.32@tcp231 [ 1400.684366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1400.771088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1400.791551] LNet: Added LNI 192.168.203.32@tcp231 [8/256/0/0] [ 1400.795635] LNet: 84225:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1401.189639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1401.217356] LNet: Removed LNI 192.168.203.32@tcp231 [ 1401.236279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1401.258807] LNet: Added LNI 192.168.203.32@tcp231 [8/256/0/0] [ 1402.346899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1402.361607] LNet: 84368:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1402.365517] LNet: 84368:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 1402.384638] LNet: Removed LNI 192.168.203.32@tcp231 [ 1402.407982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1402.477338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1402.494963] LNet: Added LNI 192.168.203.32@tcp231 [8/256/0/60] [ 1402.502604] LNet: 84376:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1402.977758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1403.035492] LNet: Removed LNI 192.168.203.32@tcp231 [ 1403.054338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1403.077061] LNet: Added LNI 192.168.203.32@tcp231 [8/256/0/60] [ 1404.359081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1404.382239] LNetError: 84520:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1404.390427] LNetError: 84520:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1404.403530] LNet: Removed LNI 192.168.203.32@tcp231 [ 1404.427543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1404.506512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1404.538822] LNet: Added LNI 192.168.203.32@tcp231 [8/256/0/180] [ 1404.543096] LNet: 84528:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1404.812774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1404.842553] LNet: Removed LNI 192.168.203.32@tcp231 [ 1404.855228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1404.874389] LNet: Added LNI 192.168.203.32@tcp231 [8/256/0/180] [ 1411.311549] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 18:21:17 (1763335277) [ 1411.415180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1411.462760] LNet: Removed LNI 192.168.203.32@tcp231 [ 1411.496035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1411.603958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1411.632348] LNet: Added LNI 192.168.203.32@tcp232 [8/256/0/180] [ 1412.448891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1412.487196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1412.517254] LNet: Removed LNI 192.168.203.32@tcp232 [ 1412.543367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1412.578160] LNet: Added LNI 192.168.203.32@tcp232 [8/256/0/180] [ 1413.320368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1413.359272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1413.389993] LNet: Removed LNI 192.168.203.32@tcp232 [ 1413.404820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1413.435374] LNet: Added LNI 192.168.203.32@tcp232 [8/256/0/180] [ 1414.147532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1414.192174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1414.227452] LNet: Removed LNI 192.168.203.32@tcp232 [ 1414.260804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1414.315348] LNet: Added LNI 192.168.203.32@tcp232 [8/256/0/180] [ 1415.011351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1415.057905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1415.082327] LNet: Removed LNI 192.168.203.32@tcp232 [ 1415.097891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1420.844783] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 18:21:27 (1763335287) [ 1421.843472] Key type .llcrypt unregistered [ 1421.849478] Key type ._llcrypt unregistered [ 1426.965260] Key type ._llcrypt registered [ 1426.970750] Key type .llcrypt registered [ 1427.164957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1427.177053] alg: No test for adler32 (adler32-zlib) [ 1428.144843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1428.198731] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1434.555493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.132@tcp [ 1441.468035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1441.497907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1441.514114] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 1443.047961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_interrupt -r 2 [ 1443.084510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e local_interrupt -r 2 [ 1443.117707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_interrupt -r 2 [ 1443.146321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e local_interrupt -r 2 [ 1443.185099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e local_interrupt -r 2 [ 1443.236300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e local_interrupt -r 2 [ 1443.309522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e local_interrupt -r 2 [ 1443.342923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e local_interrupt -r 2 [ 1443.372931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.32@tcp 192.168.203.132@tcp [ 1443.400236] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1443.645737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1443.693537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1443.721119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1443.741915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1443.769675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1443.793596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1443.818960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1443.862154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1443.876342] LNet: 87021:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1443.906657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1443.929653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1443.951650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1443.983717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1444.013506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1444.060354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1444.511709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_dropped -r 2 [ 1444.545407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e local_dropped -r 2 [ 1444.578640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_dropped -r 2 [ 1444.615339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e local_dropped -r 2 [ 1444.652530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e local_dropped -r 2 [ 1444.678868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e local_dropped -r 2 [ 1444.710187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e local_dropped -r 2 [ 1444.752263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e local_dropped -r 2 [ 1444.778357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.32@tcp 192.168.203.132@tcp [ 1444.801144] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1444.813700] LNet: 1 local NIs in recovery (showing 1): 192.168.203.32@tcp [ 1445.024572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1445.062955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1445.104028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1445.134487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1445.169759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1445.219951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1445.256962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1445.292922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1445.307625] LNet: 87141:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1445.318896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1445.345368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1445.377632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1445.409434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1445.429087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1445.454717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1445.892948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_aborted -r 2 [ 1445.940972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e local_aborted -r 2 [ 1445.972916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_aborted -r 2 [ 1446.013325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e local_aborted -r 2 [ 1446.051596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e local_aborted -r 2 [ 1446.108749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e local_aborted -r 2 [ 1446.151475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e local_aborted -r 2 [ 1446.181427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e local_aborted -r 2 [ 1446.233978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.32@tcp 192.168.203.132@tcp [ 1446.273083] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1446.506583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1446.527131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1446.552177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1446.577491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1446.603484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1446.634224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1446.671509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1446.696604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1446.722960] LNet: 87262:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1446.741667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1446.762898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1446.787530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1446.804268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1446.834507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1446.862576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1447.393396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_no_route -r 2 [ 1447.418412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e local_no_route -r 2 [ 1447.445479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_no_route -r 2 [ 1447.470919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e local_no_route -r 2 [ 1447.507553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e local_no_route -r 2 [ 1447.541609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e local_no_route -r 2 [ 1447.595647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e local_no_route -r 2 [ 1447.631475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e local_no_route -r 2 [ 1447.651947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.32@tcp 192.168.203.132@tcp [ 1447.892779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1447.957893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1448.008917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1448.054571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1448.084838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1448.133853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1448.175493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1448.212533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1448.232978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1448.270881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1448.300832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1448.326497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1448.365280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1448.405153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1448.904939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_timeout -r 2 [ 1448.935487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e local_timeout -r 2 [ 1448.969121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_timeout -r 2 [ 1448.992204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e local_timeout -r 2 [ 1449.028573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e local_timeout -r 2 [ 1449.062512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e local_timeout -r 2 [ 1449.098360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e local_timeout -r 2 [ 1449.134850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e local_timeout -r 2 [ 1449.164227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.32@tcp 192.168.203.132@tcp [ 1449.186661] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1449.192964] LNet: Skipped 1 previous similar message [ 1449.401513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1449.434669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1449.457312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1449.488567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1449.514123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1449.538787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1449.565738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1449.583966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1449.609254] LNet: 87502:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1449.613594] LNet: 87502:0:(api-ni.c:340:recovery_interval_set()) Skipped 1 previous similar message [ 1449.635679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1449.673495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1449.702682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1449.732978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1449.772221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1449.796870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1450.293369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e remote_dropped -r 2 [ 1450.327427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e remote_dropped -r 2 [ 1450.357983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e remote_dropped -r 2 [ 1450.405888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e remote_dropped -r 2 [ 1450.459571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e remote_dropped -r 2 [ 1450.523425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e remote_dropped -r 2 [ 1450.583305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e remote_dropped -r 2 [ 1450.643813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e remote_dropped -r 2 [ 1450.683224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.32@tcp 192.168.203.132@tcp [ 1450.706339] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.132@tcp [ 1450.900139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1450.950213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1450.985968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1451.024427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1451.064884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1451.084259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1451.119628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1451.144622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1451.172669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1451.195704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1451.234250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1451.268759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1451.307535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1451.337280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1455.955154] LNet: 87695:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1455.960919] LNetError: 87695:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1455.971626] LNet: Removed LNI 192.168.203.32@tcp [ 1455.981865] LNet: Removed LNI 192.168.203.32@tcp1 [ 1456.327670] Key type .llcrypt unregistered [ 1456.330308] Key type ._llcrypt unregistered [ 1469.577216] Key type ._llcrypt registered [ 1469.583231] Key type .llcrypt registered [ 1471.033439] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 18:22:17 (1763335337) [ 1471.590393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1471.615068] alg: No test for adler32 (adler32-zlib) [ 1472.565209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1472.616625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1472.643631] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1473.690581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1473.747743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.32@tcp [ 1473.813136] LNetError: 88671:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.203.32@tcp to recovery [ 1473.957371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.32@tcp [ 1474.120398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1480.053763] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 18:22:26 (1763335346) [ 1480.117320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1480.153774] LNet: 89253:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1480.160759] LNetError: 89253:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1480.182129] LNet: Removed LNI 192.168.203.32@tcp [ 1480.212103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1480.288264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1480.332414] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1480.995525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1481.019247] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 1481.873129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.32@tcp [ 1481.904064] LNetError: 89526:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.203.32@tcp to recovery [ 1482.271363] LNet: 1 local NIs in recovery (showing 1): 192.168.203.32@tcp [ 1482.273983] LNet: There was an unexpected network error while writing to 192.168.203.32: rc = -22 [ 1483.128358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.32@tcp [ 1490.873416] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 18:22:37 (1763335357) [ 1492.363595] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg332-server [ 1494.123160] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 18:22:40 (1763335360) [ 1494.197566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1494.226249] LNet: 90475:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1494.239833] LNetError: 90475:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1494.248595] LNet: Removed LNI 192.168.203.32@tcp [ 1494.265337] LNet: Removed LNI 192.168.203.32@tcp1 [ 1494.317490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1494.373919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1494.408037] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1495.202977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1495.281886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1495.324832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1495.378748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1495.408932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1495.439381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1500.607112] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 18:22:47 (1763335367) [ 1500.856465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1500.885746] LNet: 91208:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1500.893260] LNetError: 91208:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1500.898200] LNetError: 91208:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1500.915748] LNet: Removed LNI 192.168.203.32@tcp [ 1500.944856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1501.047208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1501.089696] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1506.985515] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 18:22:53 (1763335373) [ 1507.208553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1507.226781] LNet: 91911:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1507.235548] LNetError: 91911:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1507.249922] LNet: Removed LNI 192.168.203.32@tcp [ 1507.276798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1507.324479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1507.360945] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1513.399933] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 18:23:00 (1763335380) [ 1514.752492] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1516.487381] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 18:23:03 (1763335383) [ 1516.927718] LNet: 92946:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1516.937301] LNetError: 92946:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1516.959577] LNet: Removed LNI 192.168.203.32@tcp [ 1517.517321] Key type .llcrypt unregistered [ 1517.519573] Key type ._llcrypt unregistered [ 1522.419290] Key type ._llcrypt registered [ 1522.421070] Key type .llcrypt registered [ 1522.647476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1522.664915] alg: No test for adler32 (adler32-zlib) [ 1523.584278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1523.635661] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1530.056143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.132@tcp [ 1534.718390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.132@tcp [ 1535.118407] LNet: 93358:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1535.123060] LNetError: 93358:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1535.133123] LNet: Removed LNI 192.168.203.32@tcp [ 1535.564216] Key type .llcrypt unregistered [ 1535.565579] Key type ._llcrypt unregistered [ 1546.011192] Key type ._llcrypt registered [ 1546.012919] Key type .llcrypt registered [ 1547.313925] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1549.039900] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1551.186156] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 18:23:37 (1763335417) [ 1551.763643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1551.792058] alg: No test for adler32 (adler32-zlib) [ 1552.727506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1552.842977] LNet: 94441:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1553.543543] Key type .llcrypt unregistered [ 1553.546956] Key type ._llcrypt unregistered [ 1558.951538] Key type ._llcrypt registered [ 1558.953170] Key type .llcrypt registered [ 1559.138125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1559.174228] alg: No test for adler32 (adler32-zlib) [ 1560.193507] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1560.195673] LNet: 94706:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1560.250132] LNetError: 94718:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.34@tcp is being used as a gateway but routing feature is not turned on [ 1560.263750] LNetError: 94718:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.203.34@tcp has gone from up to down [ 1561.129717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1561.177484] LNet: 94764:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1561.194853] LNetError: 94764:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1562.213976] LNet: Removed LNI 192.168.203.32@tcp [ 1562.248839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1562.299361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1562.320562] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1562.326377] LNet: 94771:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1562.331321] LNet: 94771:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 1562.337964] LNetError: 94771:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.203.33@tcp is a gateway. Can not delete it [ 1562.374155] LNetError: 94771:0:(peer.c:2208:lnet_del_peer_ni()) 192.168.203.33@tcp is a router. Can not be deleted [ 1562.387260] LNetError: 94768:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.37@tcp is being used as a gateway but routing feature is not turned on [ 1562.400528] LNetError: 94768:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 1562.421369] LNetError: 94768:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.203.37@tcp has gone from up to down [ 1562.436271] LNetError: 94768:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 1562.443471] LNet: 94771:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1568.213887] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 18:23:54 (1763335434) [ 1569.525246] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg332-server [ 1570.921615] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 18:23:57 (1763335437) [ 1572.509516] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg332-server [ 1574.037713] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 18:24:00 (1763335440) [ 1575.208369] Lustre: Lustre: Build Version: 2.16.61_54_g3c0e0e6 [ 1577.143318] Key type lgssc registered [ 1578.586639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1597.304662] Key type lgssc unregistered [ 1597.591094] LNet: 97186:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1597.598486] LNetError: 97186:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1597.654343] LNet: Removed LNI 192.168.203.32@tcp [ 1598.281336] Key type .llcrypt unregistered [ 1598.289424] Key type ._llcrypt unregistered [ 1610.006106] Key type ._llcrypt registered [ 1610.007521] Key type .llcrypt registered [ 1611.211591] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 18:24:37 (1763335477) [ 1611.573996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1611.605596] alg: No test for adler32 (adler32-zlib) [ 1612.571937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1612.631470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1612.676860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1612.719719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1612.752436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1618.379285] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 18:24:44 (1763335484) [ 1618.506655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1618.538536] LNet: 98582:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1618.566753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1618.613323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1618.648079] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1619.611447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1619.656518] LNet: 98732:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1619.660290] LNetError: 98732:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1619.674596] LNet: Removed LNI 192.168.203.32@tcp [ 1619.690549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1619.732174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1619.758632] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1620.186723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1620.207801] LNetError: 98760:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1620.221727] LNet: Removed LNI 192.168.203.32@tcp [ 1620.247953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1620.309937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1620.334824] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1626.333766] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 18:24:52 (1763335492) [ 1626.450239] LNet: 99343:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1626.454765] LNet: 99343:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1626.470344] LNetError: 99343:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1626.504554] LNet: Removed LNI 192.168.203.32@tcp [ 1627.311852] Key type .llcrypt unregistered [ 1627.313739] Key type ._llcrypt unregistered [ 1632.376142] Key type ._llcrypt registered [ 1632.377934] Key type .llcrypt registered [ 1632.518445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1632.532849] alg: No test for adler32 (adler32-zlib) [ 1633.482279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1633.561306] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1633.563293] LNet: *** cfs_fail_loc=e003, val=100*** [ 1634.239474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1634.256902] LNet: *** cfs_fail_loc=e003, val=100*** [ 1634.258931] LNet: Skipped 2 previous similar messages [ 1634.281719] Ping reply received with mapping: [ 1634.283464] found metadata reply NI[192.168.203.32@tcp] [ 1634.293294] 20 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1634.295232] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1634.298853] 5a 5a 5a 5a 5a 5a 5a 5a [ 1634.416920] LNet: 99744:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1634.426592] LNetError: 99744:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1635.498862] LNet: Removed LNI 192.168.203.32@tcp [ 1636.055772] Key type .llcrypt unregistered [ 1636.061613] Key type ._llcrypt unregistered [ 1645.354357] Key type ._llcrypt registered [ 1645.355992] Key type .llcrypt registered [ 1646.759815] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 18:25:13 (1763335513) [ 1647.217416] Key type .llcrypt unregistered [ 1647.219252] Key type ._llcrypt unregistered [ 1652.553387] Key type ._llcrypt registered [ 1652.555135] Key type .llcrypt registered [ 1652.912511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1652.959167] alg: No test for adler32 (adler32-zlib) [ 1654.232147] LNetError: 100779:0:(api-ni.c:2725:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1654.679563] Key type .llcrypt unregistered [ 1654.687493] Key type ._llcrypt unregistered [ 1660.101369] Key type ._llcrypt registered [ 1660.110583] Key type .llcrypt registered [ 1661.255620] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 18:25:27 (1763335527) [ 1661.857575] Key type .llcrypt unregistered [ 1661.859606] Key type ._llcrypt unregistered [ 1666.420115] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1666.428484] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1669.080575] Key type ._llcrypt registered [ 1669.082093] Key type .llcrypt registered [ 1669.334075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1669.345863] alg: No test for adler32 (adler32-zlib) [ 1670.264775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1670.350604] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1679.986307] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 18:25:46 (1763335546) [ 1680.320658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1680.351710] LNet: 102551:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1680.364411] LNetError: 102551:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1680.385595] LNet: Removed LNI 10.1.2.3@tcp [ 1680.427165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1681.354057] Key type .llcrypt unregistered [ 1681.355737] Key type ._llcrypt unregistered [ 1686.776547] Key type ._llcrypt registered [ 1686.778428] Key type .llcrypt registered [ 1687.025430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1687.039338] alg: No test for adler32 (adler32-zlib) [ 1688.055295] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1688.112991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1688.287272] LNet: 102836:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1688.293518] LNetError: 102836:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1688.305123] LNet: Removed LNI 192.168.203.32@tcp [ 1689.067546] Key type .llcrypt unregistered [ 1689.069395] Key type ._llcrypt unregistered [ 1699.048446] Key type ._llcrypt registered [ 1699.049848] Key type .llcrypt registered [ 1700.298198] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 18:26:06 (1763335566) [ 1700.961223] Key type .llcrypt unregistered [ 1700.969763] Key type ._llcrypt unregistered [ 1706.188699] Key type ._llcrypt registered [ 1706.190428] Key type .llcrypt registered [ 1706.413377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1706.431082] alg: No test for adler32 (adler32-zlib) [ 1712.817954] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 18:26:19 (1763335579) [ 1713.077565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1713.122328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1713.201519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1713.236636] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1713.874708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1713.909906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1713.931682] LNet: Removed LNI 192.168.203.32@tcp [ 1714.142213] LNet: 104571:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1714.568717] Key type .llcrypt unregistered [ 1714.571668] Key type ._llcrypt unregistered [ 1723.161085] Key type ._llcrypt registered [ 1723.166188] Key type .llcrypt registered [ 1724.350294] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 18:26:30 (1763335590) [ 1724.662414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1724.675797] alg: No test for adler32 (adler32-zlib) [ 1725.633822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1725.719101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1725.740270] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1726.734616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1726.766421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.32@tcp [ 1726.785585] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1726.788757] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1726.791853] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1726, 0, 0 [ 1726.796222] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1726.799183] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1726.803126] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1726.806729] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1726, 0, 0 [ 1726.811111] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1726.814097] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1726.816158] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1726.818115] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1726, 0, 1776 [ 1726.820783] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1726.822841] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1726.824865] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1726.826639] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1726, 0, 1776 [ 1726.829975] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1732.568908] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 18:26:39 (1763335599) [ 1733.220449] LNet: 106107:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1733.230491] LNetError: 106107:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1733.253366] LNet: Removed LNI 192.168.203.32@tcp [ 1733.892450] Key type .llcrypt unregistered [ 1733.894658] Key type ._llcrypt unregistered [ 1738.202518] Key type ._llcrypt registered [ 1738.211868] Key type .llcrypt registered [ 1738.412795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1738.425444] alg: No test for adler32 (adler32-zlib) [ 1739.358089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1739.413915] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1744.665190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.132@tcp [ 1750.212495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1750.249392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1750.267788] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 1751.169723] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1751.290744] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1751.294829] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1751.300017] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1752.537391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1752.560333] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1763.313671] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 18:27:09 (1763335629) [ 1763.422950] LNet: 107585:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1763.434383] LNetError: 107585:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1764.448120] LNet: Removed LNI 192.168.203.32@tcp [ 1764.450201] LNetError: 107585:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1764.466696] LNet: Removed LNI 192.168.203.32@tcp1 [ 1764.482973] LNet: Removed LNI 10.1.2.3@tcp99 [ 1765.157847] Key type .llcrypt unregistered [ 1765.165861] Key type ._llcrypt unregistered [ 1771.171994] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1771.180705] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1773.774404] Key type ._llcrypt registered [ 1773.776078] Key type .llcrypt registered [ 1773.974644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1773.992387] alg: No test for adler32 (adler32-zlib) [ 1774.939803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1774.985290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1775.004744] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1775.708974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1775.720502] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1777.952429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1778.024826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1778.101667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1778.163130] LNetError: 108332:0:(peer.c:2216:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1778.201853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1778.267818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 1778.702666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1778.756973] LNet: 108352:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1778.765853] LNetError: 108352:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1778.771982] LNet: Removed LNI 192.168.203.32@tcp [ 1778.782596] LNet: Removed LNI 10.1.2.3@tcp [ 1779.336953] Key type .llcrypt unregistered [ 1779.338302] Key type ._llcrypt unregistered [ 1788.400897] Key type ._llcrypt registered [ 1788.403276] Key type .llcrypt registered [ 1789.749742] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 18:27:36 (1763335656) [ 1790.132952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1790.148476] alg: No test for adler32 (adler32-zlib) [ 1791.087061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1791.174901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1791.196665] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1791.821497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1791.888576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg332-client@tcp [ 1797.305632] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 18:27:44 (1763335664) [ 1797.377247] LNet: 109873:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1797.388385] LNetError: 109873:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1797.418695] LNet: Removed LNI 192.168.203.32@tcp [ 1798.005602] Key type .llcrypt unregistered [ 1798.009469] Key type ._llcrypt unregistered [ 1802.532275] Key type ._llcrypt registered [ 1802.534725] Key type .llcrypt registered [ 1802.713378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1802.735162] alg: No test for adler32 (adler32-zlib) [ 1803.599299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1803.627910] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1803.652210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1808.518454] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 18:27:55 (1763335675) [ 1809.460552] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1810.439704] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 18:27:57 (1763335677) [ 1811.490644] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1812.548217] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 18:27:59 (1763335679) [ 1812.706112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1812.718231] LNet: 111295:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1812.722377] LNetError: 111295:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1812.732882] LNet: Removed LNI 192.168.203.32@tcp [ 1812.746864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1812.784061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1812.801338] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1813.333601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1813.379907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1818.327623] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 18:28:05 (1763335685) [ 1818.503149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1818.519913] LNet: 112005:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1818.526236] LNetError: 112005:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1818.536311] LNet: Removed LNI 192.168.203.32@tcp [ 1818.557374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1818.590374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1818.622386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1819.048484] Key type .llcrypt unregistered [ 1819.050386] Key type ._llcrypt unregistered [ 1827.499709] Key type ._llcrypt registered [ 1827.503685] Key type .llcrypt registered [ 1828.969393] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 18:28:15 (1763335695) [ 1829.222915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1829.237069] alg: No test for adler32 (adler32-zlib) [ 1830.212547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1830.279722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1830.305742] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1831.003225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1831.093858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 1837.711617] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 18:28:24 (1763335704) [ 1837.859962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1837.878213] LNet: 113558:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1837.885128] LNetError: 113558:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1837.896767] LNet: Removed LNI 192.168.203.32@tcp [ 1837.908590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1837.938638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1837.973928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1843.396708] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 18:28:30 (1763335710) [ 1843.458920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1843.479986] LNet: 114132:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1843.500128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1843.578479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1843.838790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1843.855896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1843.896364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1844.101277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1844.132219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1844.199385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1844.237637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1844.282510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1844.529564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1844.556975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1844.592888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1844.873137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1844.895847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1849.857839] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 18:28:36 (1763335716) [ 1849.915687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1849.944938] LNet: 114763:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1849.948647] LNet: 114763:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1849.968161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1850.008354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1850.039603] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1850.652504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1850.666082] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 1851.404845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1851.456345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 1851.661203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1851.681898] LNetError: 115050:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1852.703795] LNet: Removed LNI 192.168.203.32@tcp [ 1852.705931] LNetError: 115050:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1852.718924] LNet: Removed LNI 192.168.203.32@tcp2 [ 1852.752382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1852.842820] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1852.848233] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 1852.854607] LNet: 115056:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1853.146799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 1853.283656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1853.302162] LNet: 115093:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1853.305157] LNet: 115093:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1854.370463] LNet: Removed LNI 192.168.203.32@tcp [ 1854.388166] LNet: Removed LNI 192.168.203.32@tcp2 [ 1854.430261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1854.522525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1854.542173] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1855.266967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1855.282734] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 1856.038907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 1856.073662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1856.111686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 1856.259488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1856.278582] LNetError: 115378:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1856.281690] LNetError: 115378:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1857.311184] LNet: Removed LNI 192.168.203.32@tcp [ 1858.341070] LNet: Removed LNI 192.168.203.32@tcp2 [ 1858.364986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1858.409675] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1858.416867] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 1858.421819] LNet: 115384:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1858.743836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 1858.939417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1858.958755] LNet: 115421:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1858.964873] LNet: 115421:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1858.969234] LNet: Removed LNI 192.168.203.32@tcp [ 1860.006937] LNet: Removed LNI 192.168.203.32@tcp2 [ 1860.025588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1860.074563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1860.106357] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1860.584730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1860.600177] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 1861.226089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1861.293663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 1861.507155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1861.542771] LNetError: 115701:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1861.547168] LNetError: 115701:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1861.569536] LNet: Removed LNI 192.168.203.32@tcp [ 1861.578155] LNet: Removed LNI 192.168.203.32@tcp2 [ 1861.596481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1861.649733] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1861.666415] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 1861.669426] LNet: 115707:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1862.012046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 1862.179484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1863.263171] LNet: Removed LNI 192.168.203.32@tcp [ 1863.276965] LNet: Removed LNI 192.168.203.32@tcp2 [ 1863.303281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1863.398172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1863.419024] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1864.038956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1864.051753] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 1864.768868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 1864.820156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1864.885667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 1865.140459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1866.212554] LNet: Removed LNI 192.168.203.32@tcp [ 1867.246347] LNet: Removed LNI 192.168.203.32@tcp2 [ 1867.287324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1867.346576] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1867.353152] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 1867.355814] LNet: 116034:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1867.709899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 1867.868643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1867.891786] LNet: 116072:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1867.894696] LNet: 116072:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1867.900300] LNet: Removed LNI 192.168.203.32@tcp [ 1868.968642] LNet: Removed LNI 192.168.203.32@tcp2 [ 1869.004457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1869.076856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1869.109127] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1869.637941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1869.652776] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 1870.291270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1870.312894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1870.355685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 1870.645136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1870.672985] LNetError: 116368:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1870.678797] LNetError: 116368:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1870.687383] LNet: Removed LNI 192.168.203.32@tcp [ 1870.698499] LNet: Removed LNI 192.168.203.32@tcp2 [ 1870.719465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1870.784196] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1870.789334] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 1871.093679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 1871.403387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1872.482775] LNet: Removed LNI 192.168.203.32@tcp [ 1872.494894] LNet: Removed LNI 192.168.203.32@tcp2 [ 1872.512454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1872.587429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1872.604518] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1873.132907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1873.151957] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 1873.768939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 1873.805813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1873.839484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1873.889341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 1874.171854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1875.231619] LNet: Removed LNI 192.168.203.32@tcp [ 1876.267072] LNet: Removed LNI 192.168.203.32@tcp2 [ 1876.280312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1876.323145] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1876.329622] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 1876.333654] LNet: 116734:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1876.337209] LNet: 116734:0:(api-ni.c:340:recovery_interval_set()) Skipped 1 previous similar message [ 1876.713296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 1882.355492] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1883.903678] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 18:29:10 (1763335750) [ 1884.004472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1884.035409] LNet: 117489:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1884.041730] LNet: 117489:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1884.050398] LNet: Removed LNI 192.168.203.32@tcp [ 1884.061343] LNet: Removed LNI 192.168.203.32@tcp2 [ 1884.087551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1884.336158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1884.361825] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1885.014837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1885.027110] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1885.642402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1885.655914] LNet: Removed LNI 192.168.203.32@tcp [ 1885.662530] LNet: Removed LNI 10.1.2.3@tcp [ 1885.672212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1885.711949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1885.727295] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1885.730496] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1886.136387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1886.150654] LNet: Removed LNI 192.168.203.32@tcp [ 1886.165919] LNet: Removed LNI 10.1.2.3@tcp [ 1890.350849] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 18:29:17 (1763335757) [ 1890.485552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1890.515584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1890.600403] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1890.700212] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1890.702295] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1890.705270] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1892.936811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1892.963500] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1893.488587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1893.506724] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1894.868731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 1894.891880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1894.939500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.32@tcp -m PUT -e local_timeout -r 1 [ 1894.980660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1895.025334] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1895.030930] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1895.206670] LNetError: 118432:0:(lib-move.c:4947:lnet_parse()) 192.168.203.32@tcp, src 192.168.203.32@tcp: Dropping PUT (error -108 looking up sender) [ 1896.225705] LNetError: 118889:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1896.228938] LNetError: 118889:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 1896.249225] LNet: Removed LNI 192.168.203.32@tcp [ 1896.257808] LNet: Removed LNI 10.1.2.3@tcp [ 1896.540955] Key type .llcrypt unregistered [ 1896.543130] Key type ._llcrypt unregistered [ 1903.442386] Key type ._llcrypt registered [ 1903.444368] Key type .llcrypt registered [ 1904.443670] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 18:29:31 (1763335771) [ 1904.705277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1904.719937] alg: No test for adler32 (adler32-zlib) [ 1905.642849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1905.911370] LNet: 119738:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1906.482610] Key type .llcrypt unregistered [ 1906.484423] Key type ._llcrypt unregistered [ 1910.930358] Key type ._llcrypt registered [ 1910.932095] Key type .llcrypt registered [ 1911.240166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1911.254824] alg: No test for adler32 (adler32-zlib) [ 1912.181970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1912.210951] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1917.864348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.132@tcp [ 1918.051025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1918.579229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.132@tcp [ 1918.794315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1918.823129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1918.865405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1918.902685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1918.944138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1918.982204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1919.023700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1919.057260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1919.077836] LNet: 120183:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1919.094599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1919.144840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1919.192541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1919.235450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1919.263706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1919.300916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1919.793665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1919.985645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1920.007062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1920.041856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1920.061063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1920.107821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1920.149760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1920.182922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1920.211752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1920.231124] LNet: 120300:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1920.249500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1920.279089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1920.310390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1920.340954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1920.367471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1920.390260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1920.882194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.132@tcp [ 1921.034583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1921.058090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1921.075324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1921.097402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1921.118945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1921.138469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1921.160886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1921.188745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1921.215802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1921.238461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1921.258155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1921.281577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1921.318061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1921.345283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1921.816460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1922.006348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1922.040426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1922.063638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1922.095594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1922.135333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1922.166330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1922.195716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1922.222613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1922.240134] LNet: 120535:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1922.244313] LNet: 120535:0:(api-ni.c:340:recovery_interval_set()) Skipped 1 previous similar message [ 1922.258822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1922.287252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1922.327850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1922.367492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1922.393220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1922.421605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1922.669339] LNet: 120580:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1922.673653] LNetError: 120580:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1922.686094] LNet: Removed LNI 192.168.203.32@tcp [ 1922.943525] Key type .llcrypt unregistered [ 1922.945074] Key type ._llcrypt unregistered [ 1934.411041] Key type ._llcrypt registered [ 1934.412429] Key type .llcrypt registered [ 1935.395069] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 18:30:02 (1763335802) [ 1935.612431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1935.622263] alg: No test for adler32 (adler32-zlib) [ 1936.486458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1936.804899] LNet: 121463:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1937.224708] Key type .llcrypt unregistered [ 1937.227988] Key type ._llcrypt unregistered [ 1941.421920] Key type ._llcrypt registered [ 1941.426347] Key type .llcrypt registered [ 1941.636733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1941.648198] alg: No test for adler32 (adler32-zlib) [ 1942.538175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1942.565802] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1948.292117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.132@tcp [ 1954.145104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1954.174200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1954.194893] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 1954.788794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1955.127769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.132@tcp [ 1955.277542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1955.299477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1955.325417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1955.357359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1955.382332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1955.412389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1955.444282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1955.467456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1955.479677] LNet: 122110:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1955.493511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1955.530993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1955.593235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1955.614428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1955.641920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1955.672171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1956.162498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1956.335284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1956.365740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1956.387865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1956.407330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1956.436408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1956.463137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1956.481352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1956.507467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1956.519337] LNet: 122227:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1956.531393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1956.552762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1956.578887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1956.603232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1956.630456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1956.661991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1957.146592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.132@tcp [ 1957.343463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1957.364530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1957.386409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1957.407377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1957.432337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1957.451688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1957.475236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1957.505914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1957.520704] LNet: 122344:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1957.540052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1957.560872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1957.584607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1957.606139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1957.633463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1957.664671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1958.138576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1958.285315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1958.320806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1958.357815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1958.386421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1958.416957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1958.444810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1958.466153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1958.489874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1958.517641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1958.544434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1958.579503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1958.604987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1958.632361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1958.658756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1962.076771] LNet: 122541:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1962.082375] LNetError: 122541:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1962.087171] LNet: Removed LNI 192.168.203.32@tcp [ 1962.101363] LNet: Removed LNI 192.168.203.32@tcp1 [ 1962.445904] Key type .llcrypt unregistered [ 1962.447803] Key type ._llcrypt unregistered [ 1972.685805] Key type ._llcrypt registered [ 1972.687414] Key type .llcrypt registered [ 1973.522695] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1772 sec ========== 18:30:40 (1763335840) [ 1973.880640] Key type .llcrypt unregistered [ 1973.881953] Key type ._llcrypt unregistered [ 1980.653363] Key type ._llcrypt registered [ 1980.654969] Key type .llcrypt registered [ 1980.856652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1980.864140] alg: No test for adler32 (adler32-zlib) [ 1981.778538] Lustre: Lustre: Build Version: 2.16.61_54_g3c0e0e6 [ 1981.903540] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1983.504189] Key type lgssc registered [ 1984.146169] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2025.673669] Lustre: Mounted lustre-client [ 2029.899992] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2040.369634] LustreError: 125761:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9184ca07f800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 2040.388585] LustreError: 125761:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 2040.433734] Lustre: Unmounted lustre-client [ 2059.149620] Key type lgssc unregistered [ 2059.395871] LNet: 126241:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2059.401300] LNetError: 126241:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2059.420547] LNet: Removed LNI 192.168.203.32@tcp [ 2059.944321] Key type .llcrypt unregistered [ 2059.946118] Key type ._llcrypt unregistered