[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-22) (GCC)) #7 SMP Sat Jan 18 21:01:29 EST 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-1.fc39 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 414813798 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5410-0x000f541f] [ 0.000000] RAMDISK: [mem 0xbcbf3000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5220 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D6F 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C0B 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BCB (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C7F 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D0F 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D47 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c0b-0xbffe1c7e] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c0a] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c7f-0xbffe1d0e] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d0f-0xbffe1d46] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d47-0xbffe1d6e] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 513 pages/cpu s2064384 r8192 d28672 u4194304 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2819008K/4306400K available (20483K kernel code, 12066K rwdata, 7356K rodata, 4680K init, 23504K bss, 542472K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41388 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x616/0x99a with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar [ 0.001000] ... MAX_LOCKDEP_SUBCLASSES: 8 [ 0.001000] ... MAX_LOCK_DEPTH: 48 [ 0.001000] ... MAX_LOCKDEP_KEYS: 8192 [ 0.001000] ... CLASSHASH_SIZE: 4096 [ 0.001000] ... MAX_LOCKDEP_ENTRIES: 32768 [ 0.001000] ... MAX_LOCKDEP_CHAINS: 65536 [ 0.001000] ... CHAINHASH_SIZE: 32768 [ 0.001000] memory used by lock dependency info: 4149 kB [ 0.001000] per task-struct memory footprint: 2688 bytes [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.002260] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.006993] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008007] pid_max: default: 32768 minimum: 301 [ 0.009247] LSM: Security Framework initializing [ 0.010077] Yama: becoming mindful. [ 0.011081] SELinux: Initializing. [ 0.012161] *** VALIDATE selinux *** [ 0.018436] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.021955] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.022138] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.023089] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024177] *** VALIDATE tmpfs *** [ 0.026504] *** VALIDATE proc *** [ 0.027555] *** VALIDATE cgroup *** [ 0.028006] *** VALIDATE cgroup2 *** [ 0.029257] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.030136] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.031005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.032022] Spectre V2 : User space: Vulnerable [ 0.033006] Speculative Store Bypass: Vulnerable [ 0.035857] debug: unmapping init [mem 0xffffffffb5303000-0xffffffffb530afff] [ 0.037283] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040220] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041019] ... version: 2 [ 0.041786] ... bit width: 48 [ 0.042009] ... generic registers: 4 [ 0.042702] ... value mask: 0000ffffffffffff [ 0.043008] ... max period: 00007fffffffffff [ 0.044008] ... fixed-purpose events: 3 [ 0.045008] ... event mask: 000000070000000f [ 0.046402] rcu: Hierarchical SRCU implementation. [ 0.050557] smp: Bringing up secondary CPUs ... [ 0.052470] x86: Booting SMP configuration: [ 0.053011] .... node #0, CPUs: #1 [ 0.057167] #2 [ 0.060378] #3 [ 0.062431] smp: Brought up 1 node, 4 CPUs [ 0.063021] smpboot: Max logical packages: 1 [ 0.063961] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.244632] node 0 deferred pages initialised in 178ms [ 0.247747] pgdatinit0 (35) used greatest stack depth: 14528 bytes left [ 0.251932] devtmpfs: initialized [ 0.253277] x86/mm: Memory block size: 128MB [ 0.268617] gcov: version magic: 0x41383552 [ 0.272886] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.275131] futex hash table entries: 1024 (order: 5, 131072 bytes, vmalloc) [ 0.277884] pinctrl core: initialized pinctrl subsystem [ 0.279909] [ 0.281014] ************************************************************* [ 0.282025] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.284013] ** ** [ 0.286018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.288013] ** ** [ 0.290014] ** This means that this kernel is built to expose internal ** [ 0.292013] ** IOMMU data structures, which may compromise security on ** [ 0.294014] ** your system. ** [ 0.296036] ** ** [ 0.298013] ** If you see this message and you are not debugging the ** [ 0.301037] ** kernel, report this immediately to your vendor! ** [ 0.303014] ** ** [ 0.305014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.307012] ************************************************************* [ 0.312037] NET: Registered protocol family 16 [ 0.315414] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.318076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.321114] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.326256] cpuidle: using governor menu [ 0.329192] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.332810] PCI: Using configuration type 1 for base access [ 0.335163] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.379582] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.380026] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.394157] cryptd: max_cpu_qlen set to 1000 [ 0.398150] ACPI: Added _OSI(Module Device) [ 0.399014] ACPI: Added _OSI(Processor Device) [ 0.399897] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.401017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.426210] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.436128] ACPI: Interpreter enabled [ 0.437206] ACPI: PM: (supports S0 S3 S4 S5) [ 0.438017] ACPI: Using IOAPIC for interrupt routing [ 0.439209] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.443965] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.505811] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.507050] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.510016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.513308] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.525541] acpiphp: Slot [2] registered [ 0.526348] acpiphp: Slot [5] registered [ 0.527305] acpiphp: Slot [6] registered [ 0.528313] acpiphp: Slot [3] registered [ 0.529306] acpiphp: Slot [4] registered [ 0.530445] acpiphp: Slot [7] registered [ 0.531261] acpiphp: Slot [8] registered [ 0.532292] acpiphp: Slot [9] registered [ 0.533260] acpiphp: Slot [10] registered [ 0.534263] acpiphp: Slot [11] registered [ 0.535278] acpiphp: Slot [12] registered [ 0.536321] acpiphp: Slot [13] registered [ 0.537233] acpiphp: Slot [14] registered [ 0.538307] acpiphp: Slot [15] registered [ 0.539329] acpiphp: Slot [16] registered [ 0.540282] acpiphp: Slot [17] registered [ 0.541258] acpiphp: Slot [18] registered [ 0.542346] acpiphp: Slot [19] registered [ 0.543276] acpiphp: Slot [20] registered [ 0.544294] acpiphp: Slot [21] registered [ 0.545248] acpiphp: Slot [22] registered [ 0.546118] acpiphp: Slot [23] registered [ 0.547030] acpiphp: Slot [24] registered [ 0.548382] acpiphp: Slot [25] registered [ 0.549318] acpiphp: Slot [26] registered [ 0.550293] acpiphp: Slot [27] registered [ 0.551335] acpiphp: Slot [28] registered [ 0.552336] acpiphp: Slot [29] registered [ 0.553328] acpiphp: Slot [30] registered [ 0.554450] acpiphp: Slot [31] registered [ 0.555191] PCI host bridge to bus 0000:00 [ 0.556042] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.557104] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.558033] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.559030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.560025] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.561034] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.562445] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.564978] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.566993] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.570729] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.572585] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.573011] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.574010] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.575027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.577290] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.578818] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.579021] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.581316] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.583011] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.590018] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.592012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.613405] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.616030] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.619033] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.626048] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.640874] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.643027] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.646023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.652015] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.671621] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.673140] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.675358] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.677275] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.679088] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.686584] iommu: Default domain type: Passthrough [ 0.690000] SCSI subsystem initialized [ 0.690000] ACPI: bus type USB registered [ 0.692388] usbcore: registered new interface driver usbfs [ 0.694203] usbcore: registered new interface driver hub [ 0.696154] usbcore: registered new device driver usb [ 0.697733] pps_core: LinuxPPS API ver. 1 registered [ 0.699016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.702178] PTP clock support registered [ 0.704649] EDAC MC: Ver: 3.0.0 [ 0.706629] PCI: Using ACPI for IRQ routing [ 0.710298] NetLabel: Initializing [ 0.711021] NetLabel: domain hash size = 128 [ 0.712018] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.714441] NetLabel: unlabeled traffic allowed by default [ 0.717262] vgaarb: loaded [ 0.720257] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.721017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.726945] clocksource: Switched to clocksource kvm-clock [ 1.033703] VFS: Disk quotas dquot_6.6.0 [ 1.035324] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.038192] *** VALIDATE ramfs *** [ 1.039576] *** VALIDATE hugetlbfs *** [ 1.042029] pnp: PnP ACPI init [ 1.052704] pnp: PnP ACPI: found 6 devices [ 1.092142] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.094322] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.096429] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.098835] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.101201] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.103574] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 1.107355] NET: Registered protocol family 2 [ 1.110186] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.116546] tcp_listen_portaddr_hash hash table entries: 4096 (order: 6, 360448 bytes, vmalloc) [ 1.119711] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.125491] TCP bind hash table entries: 65536 (order: 10, 5242880 bytes, vmalloc) [ 1.133072] TCP: Hash tables configured (established 65536 bind 65536) [ 1.136544] MPTCP token hash table entries: 8192 (order: 7, 786432 bytes, vmalloc) [ 1.140271] UDP hash table entries: 4096 (order: 7, 786432 bytes, vmalloc) [ 1.143271] UDP-Lite hash table entries: 4096 (order: 7, 786432 bytes, vmalloc) [ 1.145736] NET: Registered protocol family 1 [ 1.149133] RPC: Registered named UNIX socket transport module. [ 1.151248] RPC: Registered udp transport module. [ 1.152883] RPC: Registered tcp transport module. [ 1.154752] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.156985] NET: Registered protocol family 44 [ 1.158541] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.160633] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.162641] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.164773] PCI: CLS 0 bytes, default 64 [ 1.166860] Unpacking initramfs... [ 3.400056] debug: unmapping init [mem 0xffff99847cbf3000-0xffff99847ffcffff] [ 3.404266] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.406257] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.408986] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.425210] cryptomgr_test (65) used greatest stack depth: 14248 bytes left [ 4.482942] Initialise system trusted keyrings [ 4.484747] Key type blacklist registered [ 4.486822] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.566294] zbud: loaded [ 4.587847] *** VALIDATE nfs *** [ 4.589059] *** VALIDATE nfs4 *** [ 4.590986] pstore: using deflate compression [ 4.598132] Platform Keyring initialized [ 4.605631] cryptomgr_test (73) used greatest stack depth: 14024 bytes left [ 4.668247] cryptomgr_test (86) used greatest stack depth: 13800 bytes left [ 4.717958] cryptomgr_test (94) used greatest stack depth: 13640 bytes left [ 4.934231] NET: Registered protocol family 38 [ 4.937506] Key type asymmetric registered [ 4.942396] Asymmetric key parser 'x509' registered [ 4.945250] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.950216] io scheduler mq-deadline registered [ 4.952629] io scheduler kyber registered [ 4.955777] io scheduler bfq registered [ 4.958441] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.966139] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.972207] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.979665] ACPI: Power Button [PWRF] [ 7.022067] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 9.041658] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 12.507735] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 12.603001] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 12.711571] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 12.802072] Non-volatile memory driver v1.3 [ 12.814981] Linux agpgart interface v0.103 [ 13.360505] virtio_blk virtio1: [vda] 131896 512-byte logical blocks (67.5 MB/64.4 MiB) [ 13.369302] vda: detected capacity change from 0 to 67530752 [ 13.455359] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 13.463845] vdb: detected capacity change from 0 to 1073741824 [ 13.533995] libphy: Fixed MDIO Bus: probed [ 13.588698] usbcore: registered new interface driver usbserial_generic [ 13.602853] usbserial: USB Serial support registered for generic [ 13.616100] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 13.632823] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 13.641535] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 13.653871] mousedev: PS/2 mouse device common for all mice [ 13.678025] rtc_cmos 00:05: RTC can wake from S4 [ 13.708255] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 13.720922] rtc_cmos 00:05: registered as rtc0 [ 13.734503] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 13.753366] intel_pstate: CPU model not supported [ 13.766560] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 13.810834] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 13.816836] hid: raw HID events driver (C) Jiri Kosina [ 13.838256] usbcore: registered new interface driver usbhid [ 13.842472] usbhid: USB HID core driver [ 13.851735] drop_monitor: Initializing network drop monitor service [ 13.860293] Initializing XFRM netlink socket [ 13.879538] NET: Registered protocol family 10 [ 13.927464] Segment Routing with IPv6 [ 13.941018] NET: Registered protocol family 17 [ 13.955092] mpls_gso: MPLS GSO support [ 14.000412] RAS: Correctable Errors collector initialized. [ 14.012742] AVX version of gcm_enc/dec engaged. [ 14.013906] AES CTR mode by8 optimization enabled [ 15.034888] sched_clock: Marking stable (15034151314, 0)->(15792302281, -758150967) [ 15.060863] registered taskstats version 1 [ 15.075379] Loading compiled-in X.509 certificates [ 15.092527] zswap: loaded using pool lzo/zbud [ 15.332124] Key type big_key registered [ 15.438195] Key type encrypted registered [ 15.442143] ima: No TPM chip found, activating TPM-bypass! [ 15.447848] ima: Allocated hash algorithm: sha1 [ 15.452677] ima: No architecture policies found [ 15.458408] evm: Initialising EVM extended attributes: [ 15.463444] evm: security.selinux [ 15.466276] evm: security.ima [ 15.467391] evm: security.capability [ 15.473737] evm: HMAC attrs: 0x1 [ 15.503712] rtc_cmos 00:05: setting system clock to 2025-04-01 07:33:14 UTC (1743492794) [ 15.600803] debug: unmapping init [mem 0xffffffffb6803000-0xffffffffb69fffff] [ 15.623861] debug: unmapping init [mem 0xffffffffb4e71000-0xffffffffb5302fff] [ 15.632421] Write protecting the kernel read-only data: 30720k [ 15.642465] debug: unmapping init [mem 0xffffffffb3403000-0xffffffffb35fffff] [ 15.650269] debug: unmapping init [mem 0xffffffffb3d2f000-0xffffffffb3dfffff] [ 16.116694] systemd[1]: systemd 239 (239-82.el8_10.3) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.150179] systemd[1]: Detected virtualization kvm. [ 16.163248] systemd[1]: Detected architecture x86-64. [ 16.173974] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 16.346395] systemd[1]: No hostname configured. [ 16.377811] systemd[1]: Set hostname to . [ 16.380525] random: systemd: uninitialized urandom read (16 bytes read) [ 16.383029] systemd[1]: Initializing machine ID from random generator. [ 17.079553] random: ln: uninitialized urandom read (6 bytes read) [ 17.982932] dracut-nbd-gene (195) used greatest stack depth: 13608 bytes left [ 18.188596] random: systemd: uninitialized urandom read (16 bytes read) [ 18.195811] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 18.216612] random: systemd: uninitialized urandom read (16 bytes read) [ 18.220953] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 18.243444] random: systemd: uninitialized urandom read (16 bytes read) [ 18.249550] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. Starting Setup Virtual Console... Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Sockets. [ OK ] Reached target Swap. Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. [ 20.386038] mount (252) used greatest stack depth: 13184 bytes left [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ 21.194619] systemd[1]: Started Apply Kernel Variables. [ OK ] Started Apply Kernel Variables. [ 21.294617] systemd[1]: Started Create Volatile Files and Directories. [ OK ] Started Create Volatile Files and Directories. [ 21.567962] systemd[1]: Starting Create Static Device Nodes in /dev... Starting Create Static Device Nodes in /dev... [ 21.734426] systemd[1]: Starting dracut cmdline hook... Starting dracut cmdline hook... [ 22.235040] systemd[1]: Started Create Static Device Nodes in /dev. [ OK ] Started Create Static Device Nodes in /dev. [ 22.758647] systemd[1]: Started Journal Service. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 25.058649] device-mapper: uevent: version 1.0.3 [ 25.069767] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 31.785791] virtio_net virtio0 ens2: renamed from eth0 [ 32.792523] scsi host0: ata_piix [ 32.934088] scsi host1: ata_piix [ 32.935922] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 32.956961] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 38.733699] random: crng init done [ 38.734997] random: 5 urandom warning(s) missed due to ratelimiting [ 39.385249] ip (531) used greatest stack depth: 11496 bytes left [ 44.574863] dracut-initqueue[593]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 49.064076] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 57.709842] printk: systemd: 20 output lines suppressed due to ratelimiting [ 59.650752] SELinux: Disabled at runtime. [ 59.901251] systemd[1]: systemd 239 (239-82.el8_10.3) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 59.934600] systemd[1]: Detected virtualization kvm. [ 59.937214] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 63.370458] systemd[1]: initrd-switch-root.service: Succeeded. [ 63.389370] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 63.436298] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 63.441338] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 63.445839] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 63.529454] systemd[1]: Starting Journal Service... Starting Journal Service... [ 63.565533] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ 64.211191] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. [ 64.905996] systemd[1]: Starting udev Coldplug all Devices... Starting udev Coldplug all Devices... [ 64.988305] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ 65.035775] systemd[1]: Reached target RPC Port Mapper. [ OK ] Reached target RPC Port Mapper. [ 65.068734] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ 65.219123] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [ 65.223836] systemd[1]: Failed to set up automount Arbitrary Executable File Formats File System Automount Point. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 65.289705] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ 65.351682] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 65.392777] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ 65.464484] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... Mounting POSIX Message Queue File System... [ OK ] Reached target Paths. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 69.802728] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 73.017861] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 74.484479] input: PC Speaker as /devices/platform/pcspkr/input/input5 [* ] A start job is running for Configur…only root support (14s / no limit) [** ] A start job is running for Configur…only root support (14s / no limit) [*** ] A start job is running for Configur…only root support (15s / no limit)[ 78.762528] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ *** ] A start job is running for Configur…only root support (16s / no limit) [ *** ] A start job is running for Configur…only root support (16s / no limit)[ 79.557643] EDAC sbridge: Ver: 1.1.2 [ ***] A start job is running for Configur…only root support (17s / no limit) [ **] A start job is running for Configur…only root support (17s / no limit) [ *] A start job is running for Configur…only root support (18s / no limit) [ **] A start job is running for Configur…only root support (18s / no limit) [ ***] A start job is running for Configur…only root support (19s / no limit) [ *** ] A start job is running for Configur…only root support (19s / no limit) [ *** ] A start job is running for Configur…only root support (20s / no limit) [*** ] A start job is running for Configur…only root support (21s / no limit) [** ] A start job is running for Configur…only root support (22s / no limit) [* ] A start job is running for Configur…only root support (23s / no limit) [** ] A start job is running for Configur…only root support (23s / no limit)[ 87.086588] hrtimer: interrupt took 8581656 ns [*** ] A start job is running for Configur…only root support (24s / no limit) [ *** ] A start job is running for Configur…only root support (24s / no limit) [ *** ] A start job is running for Configur…only root support (25s / no limit) [ ***] A start job is running for Configur…only root support (25s / no limit) [ **] A start job is running for Configur…only root support (26s / no limit) [ *] A start job is running for Configur…only root support (26s / no limit) [ **] A start job is running for Configur…only root support (27s / no limit) [ ***] A start job is running for Configur…only root support (27s / no limit) [ *** ] A start job is running for Configur…only root support (28s / no limit) [ *** ] A start job is running for Configur…only root support (30s / no limit)[ 93.878577] Key type dns_resolver registered [*** ] A start job is running for Configur…only root support (31s / no limit) [** ] A start job is running for Configur…only root support (31s / no limit) [* ] A start job is running for Configur…only root support (32s / no limit) [** ] A start job is running for Configur…only root support (32s / no limit) [*** ] A start job is running for Configur…only root support (33s / no limit)[ 96.639851] NFS: Registering the id_resolver key type [ 96.642046] Key type id_resolver registered [ 96.651825] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (33s / no limit) [ *** ] A start job is running for Configur…only root support (34s / no limit) [ ***] A start job is running for Configur…only root support (35s / no limit) [ **] A start job is running for Configur…only root support (35s / no limit) [ *] A start job is running for Configur…only root support (36s / no limit) [ **] A start job is running for Configur…only root support (37s / no limit)[ 100.169562] mount.nfs (969) used greatest stack depth: 10736 bytes left [ ***] A start job is running for Configur…only root support (37s / no limit) [ *** ] A start job is running for Configur…only root support (38s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ *** ] A start job is running for Rebuild …amic Linker Cache (46s / no limit) [*** ] A start job is running for Rebuild …amic Linker Cache (47s / no limit) [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg632-client login: [ 304.309960] libcfs: loading out-of-tree module taints kernel. [ 304.591260] Key type ._llcrypt registered [ 304.592913] Key type .llcrypt registered [ 306.013204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 306.030976] alg: No test for adler32 (adler32-zlib) [ 307.950637] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 309.123470] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 309.133788] LNet: Accept secure, port 988 [ 311.447394] Key type lgssc registered [ 315.955561] Lustre: Echo OBD driver; http://www.lustre.org/ [ 598.175346] Lustre: Mounted lustre-client [ 608.395243] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 623.583473] Lustre: lustre-OST0000-osc-ffff9984e0729000: disconnect after 22s idle [ 637.978666] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing check_logdir /tmp/testlogs/ [ 647.517436] Lustre: DEBUG MARKER: oleg632-client.virtnet: executing yml_node [ 660.010734] Lustre: DEBUG MARKER: Client: 2.16.52.73 [ 666.202653] Lustre: DEBUG MARKER: MDS: 2.16.52.73 [ 673.468699] Lustre: DEBUG MARKER: OSS: 2.16.52.73 [ 678.162486] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Apr 1 03:44:13 EDT 2025 [ 716.691858] Lustre: Unmounted lustre-client [ 757.902972] Key type lgssc unregistered [ 758.796581] LNet: 8155:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 759.850787] LNet: Removed LNI 192.168.206.32@tcp [ 762.796807] Key type .llcrypt unregistered [ 762.798875] Key type ._llcrypt unregistered [ 835.137936] Key type ._llcrypt registered [ 835.139511] Key type .llcrypt registered [ 835.761639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 835.817686] alg: No test for adler32 (adler32-zlib) [ 837.181491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 837.365930] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 837.384174] LNet: Accept secure, port 988 [ 837.432297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 848.052771] Lustre: DEBUG MARKER: excepting tests: 253 254 [ 848.575131] LNet: 9035:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 848.594336] LNet: Removed LNI 192.168.206.32@tcp [ 850.329156] Key type .llcrypt unregistered [ 850.330551] Key type ._llcrypt unregistered [ 867.248629] Key type ._llcrypt registered [ 867.250230] Key type .llcrypt registered [ 871.152200] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 03:47:25 (1743493645) [ 872.433732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 872.496861] alg: No test for adler32 (adler32-zlib) [ 873.858700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 874.148276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 874.235476] LNet: 9444:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 874.298690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 874.437227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 874.480451] LNet: 9451:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 893.514748] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 03:47:48 (1743493668) [ 893.811628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 893.881734] LNet: 10026:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 893.973165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 894.274790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 911.567577] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 03:48:07 (1743493687) [ 911.958944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 912.042784] LNet: 10619:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 912.176572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 912.417911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 928.930893] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 03:48:25 (1743493705) [ 929.696628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 929.770798] LNet: 11210:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 929.848633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 930.104606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 948.575098] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 03:48:43 (1743493723) [ 948.894892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 948.977315] LNet: 11792:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 949.077636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 949.357834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 950.263081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 967.786505] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 03:49:03 (1743493743) [ 968.333273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 968.434758] LNet: 12388:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 968.608590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 968.886246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 969.830439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 971.056980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 988.999738] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 03:49:23 (1743493763) [ 989.831991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 989.912682] LNet: 12989:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 990.018527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 990.365565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi [ 1007.994544] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 03:49:43 (1743493783) [ 1008.527391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1008.720082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1008.945746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 1009.102929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 1009.897279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 1010.024419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 1011.053866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 1011.192326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 1011.749579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 1011.834850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 1012.305491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 1012.405334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 1013.224922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 1013.395068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 1014.316538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 1014.425498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 1015.380459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 1015.557783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 1016.545288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 1016.632502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 1017.412319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi [ 1017.508989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 1018.497575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 1018.614180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 1039.183104] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 03:50:14 (1743493814) [ 1040.335615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1040.434887] LNet: 14245:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1040.444533] LNet: 14245:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1040.556644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1040.866173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 1040.971424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 1058.354050] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 03:50:33 (1743493833) [ 1058.921197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1059.047817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1059.401775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 1059.552087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 1077.154796] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 03:50:52 (1743493852) [ 1077.824827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1077.938358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1078.265418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 1078.426193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 1094.811450] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 03:51:10 (1743493870) [ 1095.447657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1095.598739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1095.986584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 1096.136687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 1114.483792] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 03:51:29 (1743493889) [ 1115.010159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1115.078432] LNet: 16591:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1115.089370] LNet: 16591:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1115.226198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1115.591468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 1115.714271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 1130.849244] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 03:51:46 (1743493906) [ 1131.473433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1131.659687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1131.994657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 1132.096572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 1148.456101] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 03:52:04 (1743493924) [ 1148.855163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1149.044606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1149.263146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1149.903363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 1167.216411] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 03:52:22 (1743493942) [ 1167.784400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1167.883423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1168.650902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1169.363214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 1185.893674] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 03:52:41 (1743493961) [ 1186.268086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1186.465167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1186.844780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1187.448784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 1206.147265] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 03:53:00 (1743493980) [ 1206.398182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1206.639528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1208.242515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1208.962754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 1225.365115] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 03:53:20 (1743494000) [ 1225.982252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1226.134938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1226.500649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1226.623915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 1242.319546] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 03:53:37 (1743494017) [ 1242.639890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1242.847456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1243.183979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1243.457258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 1260.105704] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 03:53:55 (1743494035) [ 1260.486767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1260.550131] LNet: 21460:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1260.565470] LNet: 21460:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 1260.671542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1260.953803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 1279.228840] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 03:54:14 (1743494054) [ 1279.615362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1279.867693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1280.159658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 1298.785826] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 03:54:34 (1743494074) [ 1299.287652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1299.539695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1299.883211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 1300.804460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 1319.595800] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 03:54:54 (1743494094) [ 1319.953746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1320.096467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1320.370045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 1320.515568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 1337.620470] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 03:55:13 (1743494113) [ 1338.329632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1338.449664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1338.702990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 1338.777611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 1355.222143] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 03:55:30 (1743494130) [ 1355.570948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1355.718151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1356.113699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 1356.200226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 1373.549734] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 03:55:48 (1743494148) [ 1373.863188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1374.001590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1374.270192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 1374.331890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 1394.474931] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 03:56:08 (1743494168) [ 1395.147903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1395.348046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1395.615145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 1395.848804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 1395.926759] LNetError: 25596:0:(peer.c:2185:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1396.097947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 1396.222876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 1415.408930] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 03:56:30 (1743494190) [ 1416.238312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1416.343344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1416.659289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.zQORnNlSQP [ 1416.946355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.zQORnNlSQP [ 1417.159949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.zQORnNlSQP [ 1433.868749] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 03:56:49 (1743494209) [ 1434.548296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1434.668437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1434.885781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 1434.990335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 1435.162893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 1452.598279] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 03:57:07 (1743494227) [ 1452.847985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1453.091667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1453.372842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 1453.460581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 1453.540638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 1453.619662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 1453.719452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 1453.821814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 1453.887477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 1453.991693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 1454.156502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 1454.241762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 1454.340430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 1454.409916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 1454.506496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 1454.691143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 1454.822758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 1454.921966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 1454.989931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 1455.063830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 1455.180428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 1455.238229] LNetError: 27403:0:(router.c:632:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 1472.044235] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 03:57:27 (1743494247) [ 1472.615344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1472.836867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1473.099464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1490.736580] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 03:57:45 (1743494265) [ 1496.447194] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg632-server [ 1501.715154] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 03:57:56 (1743494276) [ 1502.173102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1502.322844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1502.539710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1502.605757] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1502.610218] LNet: Accept secure, port 988 [ 1507.581124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.206.[33-37]@tcp [ 1507.645844] LNet: 29122:0:(router.c:737:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1507.759543] LNetError: 29124:0:(lib-lnet.h:1313:lnet_set_route_aliveness()) route to tcp1 through 192.168.206.36@tcp has gone from up to down [ 1507.771665] LNetError: 29123:0:(router.c:334:lnet_is_route_alive()) peer 192.168.206.33@tcp is being used as a gateway but routing feature is not turned on [ 1507.797949] LNetError: 28914:0:(router.c:400:lnet_router_discovery_ping_reply()) Peer 192.168.206.33@tcp is being used as a gateway but routing feature is not turned on [ 1526.718973] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 03:58:21 (1743494301) [ 1527.419804] LNetError: 29717:0:(router.c:334:lnet_is_route_alive()) peer 192.168.206.33@tcp is being used as a gateway but routing feature is not turned on [ 1532.289134] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg632-server [ 1536.825634] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 03:58:32 (1743494312) [ 1537.512745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1537.633902] LNet: 30060:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1537.641063] LNet: 30060:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 12 previous similar messages [ 1538.729200] LNet: Removed LNI 192.168.206.32@tcp [ 1538.873385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1539.107934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1539.181464] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1539.185647] LNet: Accept secure, port 988 [ 1541.345779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.206.[33-38/2]@tcp [ 1541.431846] LNet: 30174:0:(router.c:737:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1541.452677] LNet: 30174:0:(router.c:737:lnet_add_route()) Skipped 4 previous similar messages [ 1541.472966] LNetError: 30068:0:(lib-lnet.h:1313:lnet_set_route_aliveness()) route to tcp103 through 192.168.206.33@tcp has gone from up to down [ 1541.499591] LNetError: 30068:0:(lib-lnet.h:1313:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 1541.517933] LNetError: 30064:0:(router.c:400:lnet_router_discovery_ping_reply()) Peer 192.168.206.35@tcp is being used as a gateway but routing feature is not turned on [ 1541.549711] LNetError: 30064:0:(router.c:400:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 1541.578168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.206.[33-38/2]@tcp [ 1559.270178] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 03:58:54 (1743494334) [ 1559.586762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1560.686943] LNet: Removed LNI 192.168.206.32@tcp [ 1560.843392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1561.146950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 1561.309731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1561.502568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1561.661325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1561.812935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1561.915359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1562.141676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 1562.497867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1562.687489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1562.937923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1563.207972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1563.391768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1563.655625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 1564.107449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1564.290866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1564.672574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1565.019079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1565.212685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1565.458475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 1565.682492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1565.792418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1566.006536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1566.382390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1566.517659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1566.826219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1567.068953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1567.197226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1567.419777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1567.677124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1567.765544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1567.982373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 1568.295235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1568.409677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1568.575665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1585.736761] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 03:59:21 (1743494361) [ 1590.283714] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg632-server [ 1595.207909] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 03:59:30 (1743494370) [ 1599.774956] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg632-server [ 1603.860672] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 03:59:39 (1743494379) [ 1606.496176] Key type .llcrypt unregistered [ 1606.498086] Key type ._llcrypt unregistered [ 1625.414782] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1626.154118] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1626.179348] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1626.183383] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1626.953880] Key type ._llcrypt registered [ 1626.956605] Key type .llcrypt registered [ 1627.639296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1627.686376] alg: No test for adler32 (adler32-zlib) [ 1628.920340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1629.100935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1629.196635] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1629.202265] LNet: Accept secure, port 988 [ 1631.636964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1631.701482] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1639.421199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 1639.494493] LNet: Removed LNI 10.1.2.3@tcp [ 1639.932613] LNet: 32865:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1639.951967] LNet: Removed LNI 192.168.206.32@tcp [ 1642.372521] Key type .llcrypt unregistered [ 1642.374320] Key type ._llcrypt unregistered [ 1674.688504] Key type ._llcrypt registered [ 1674.690175] Key type .llcrypt registered [ 1678.130297] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 04:00:53 (1743494453) [ 1679.427869] Key type .llcrypt unregistered [ 1679.431171] Key type ._llcrypt unregistered [ 1697.711684] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1698.404752] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1698.431359] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1698.437595] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1699.703357] Key type ._llcrypt registered [ 1699.704755] Key type .llcrypt registered [ 1700.202780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1700.247917] alg: No test for adler32 (adler32-zlib) [ 1701.631052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1701.837787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1701.942564] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1701.952657] LNet: Accept secure, port 988 [ 1704.570617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1704.633202] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1712.546388] LNet: 34340:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1712.560774] LNet: Removed LNI 192.168.206.32@tcp [ 1712.583063] LNet: Removed LNI 10.1.2.3@tcp [ 1714.434447] Key type .llcrypt unregistered [ 1714.449624] Key type ._llcrypt unregistered [ 1746.330363] Key type ._llcrypt registered [ 1746.331975] Key type .llcrypt registered [ 1749.869667] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 04:02:04 (1743494524) [ 1751.568513] Key type .llcrypt unregistered [ 1751.570250] Key type ._llcrypt unregistered [ 1767.319840] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1768.267398] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1768.292250] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1768.312299] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1769.337467] Key type ._llcrypt registered [ 1769.339033] Key type .llcrypt registered [ 1769.782452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1769.843529] alg: No test for adler32 (adler32-zlib) [ 1771.192770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1771.350477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 1771.431154] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 1771.434336] LNet: Accept secure, port 988 [ 1777.231087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1777.317399] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1783.955354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 1783.988491] LNet: Removed LNI 10.1.2.3@tcp [ 1784.023512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 1784.077120] LNet: Removed LNI 10.1.2.31@tcp [ 1784.127309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1784.162458] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1786.167749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 1786.206534] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 1789.005989] LNet: 36224:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1789.016896] LNet: Removed LNI 10.1.2.3@tcp [ 1789.034471] LNet: Removed LNI 10.1.2.31@tcp [ 1791.198220] Key type .llcrypt unregistered [ 1791.202109] Key type ._llcrypt unregistered [ 1824.725867] Key type ._llcrypt registered [ 1824.735572] Key type .llcrypt registered [ 1828.241078] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 04:03:23 (1743494603) [ 1829.715307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1829.782888] alg: No test for adler32 (adler32-zlib) [ 1831.194528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1831.515530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1831.662716] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1831.666212] LNet: Accept secure, port 988 [ 1834.411071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1834.473715] LNet: 37176:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1834.486586] LNet: Removed LNI 192.168.206.32@tcp [ 1834.586164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1834.756580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.206.32@tcp [ 1834.833445] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1834.844616] LNet: Accept secure, port 988 [ 1837.393030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.206.32@tcp [ 1837.504317] LNet: Removed LNI 192.168.206.32@tcp [ 1837.697540] LNet: 37308:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1839.674056] Key type .llcrypt unregistered [ 1839.675620] Key type ._llcrypt unregistered [ 1873.255422] Key type ._llcrypt registered [ 1873.257670] Key type .llcrypt registered [ 1877.141433] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 04:04:11 (1743494651) [ 1883.115865] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg632-server [ 1887.546153] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 04:04:23 (1743494663) [ 1888.982964] Key type .llcrypt unregistered [ 1888.984676] Key type ._llcrypt unregistered [ 1905.090475] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1905.678552] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1905.695609] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1905.698266] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1907.266825] Key type ._llcrypt registered [ 1907.268499] Key type .llcrypt registered [ 1907.878091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1907.934707] alg: No test for adler32 (adler32-zlib) [ 1909.390494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1909.669121] LNet: Added LNI 192.168.206.32@tcp1 [8/256/0/180] [ 1909.676370] LNet: Accept secure, port 988 [ 1912.255895] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 1919.434296] LNet: 39114:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1919.446831] LNet: Removed LNI 192.168.206.32@tcp1 [ 1919.474635] LNet: Removed LNI 10.1.2.3@tcp2 [ 1922.044578] Key type .llcrypt unregistered [ 1922.049406] Key type ._llcrypt unregistered [ 1959.708592] Key type ._llcrypt registered [ 1959.710581] Key type .llcrypt registered [ 1963.618422] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 04:05:38 (1743494738) [ 1964.726332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1964.756187] alg: No test for adler32 (adler32-zlib) [ 1966.036168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1966.298058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1966.464501] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 1966.480336] LNet: Accept secure, port 988 [ 1969.192422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.206.32 [ 1969.276513] LNet: Removed LNI 192.168.206.32@tcp [ 1986.434621] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 04:06:01 (1743494761) [ 1987.092666] LNet: 40648:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1987.119492] LNet: Removed LNI 192.168.206.32@tcp [ 1989.204519] Key type .llcrypt unregistered [ 1989.206294] Key type ._llcrypt unregistered [ 2005.893334] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2006.908784] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2006.924413] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2006.932860] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2007.745778] Key type ._llcrypt registered [ 2007.750529] Key type .llcrypt registered [ 2008.396437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2008.437700] alg: No test for adler32 (adler32-zlib) [ 2009.866667] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2009.871132] LNet: Accept secure, port 988 [ 2028.159083] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 04:06:42 (1743494802) [ 2028.486928] LNet: 41511:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2028.520519] LNet: Removed LNI 10.1.2.3@tcp [ 2031.035939] Key type .llcrypt unregistered [ 2031.037447] Key type ._llcrypt unregistered [ 2048.241848] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2049.005380] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2049.012426] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2049.030456] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2049.953621] Key type ._llcrypt registered [ 2049.973865] Key type .llcrypt registered [ 2050.593598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2050.612557] alg: No test for adler32 (adler32-zlib) [ 2052.110785] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2052.117163] LNet: Accept secure, port 988 [ 2068.869981] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 04:07:24 (1743494844) [ 2069.094359] LNet: 42382:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2069.126150] LNet: Removed LNI 10.1.2.3@tcp [ 2071.743614] Key type .llcrypt unregistered [ 2071.745328] Key type ._llcrypt unregistered [ 2090.432448] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2091.172257] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2091.180352] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2091.185479] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2091.962041] Key type ._llcrypt registered [ 2091.966953] Key type .llcrypt registered [ 2092.485956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2092.514565] alg: No test for adler32 (adler32-zlib) [ 2093.858370] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2093.862141] LNet: Accept secure, port 988 [ 2110.399464] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 04:08:05 (1743494885) [ 2111.047358] LNet: 43260:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2111.068945] LNet: Removed LNI 10.1.2.3@tcp [ 2113.086711] Key type .llcrypt unregistered [ 2113.088393] Key type ._llcrypt unregistered [ 2129.285706] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2130.202920] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2130.209864] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2130.225523] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2130.993110] Key type ._llcrypt registered [ 2130.994668] Key type .llcrypt registered [ 2131.702928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2131.757037] alg: No test for adler32 (adler32-zlib) [ 2133.133487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2133.349720] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2133.356927] LNet: Accept secure, port 988 [ 2156.883367] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 04:08:52 (1743494932) [ 2158.549486] LNet: 44462:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2158.565219] LNet: Removed LNI 10.1.2.3@tcp [ 2160.626457] Key type .llcrypt unregistered [ 2160.628140] Key type ._llcrypt unregistered [ 2177.532826] Key type ._llcrypt registered [ 2177.534640] Key type .llcrypt registered [ 2178.311916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2178.342157] alg: No test for adler32 (adler32-zlib) [ 2179.782138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2179.907723] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 2179.922424] LNet: Accept secure, port 988 [ 2195.801564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.132@tcp [ 2197.275693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.132@tcp [ 2197.349405] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 2198.156633] LNet: 44857:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2198.467438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2198.639887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2199.599772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.132@tcp [ 2199.718075] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 2200.509931] LNet: 44919:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2200.794086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2200.909903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2202.045833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.132@tcp [ 2202.107923] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 2202.778472] LNet: 44981:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2203.002635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2203.088288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2204.112945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.132@tcp [ 2204.204437] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 2205.077963] LNet: 45044:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2205.367585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2205.483155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2206.498803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.132@tcp [ 2207.642388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2207.719550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2208.911803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.132@tcp [ 2208.987415] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 2208.994991] LNet: Skipped 1 previous similar message [ 2209.908883] LNet: 45168:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2209.915024] LNet: 45168:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 2210.243896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2210.355415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2211.176625] LNet: 45204:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2212.203888] LNet: Removed LNI 192.168.206.32@tcp [ 2213.480890] Key type .llcrypt unregistered [ 2213.482501] Key type ._llcrypt unregistered [ 2258.390107] Key type ._llcrypt registered [ 2258.391589] Key type .llcrypt registered [ 2261.464202] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 04:10:36 (1743495036) [ 2263.946387] Key type .llcrypt unregistered [ 2263.951740] Key type ._llcrypt unregistered [ 2280.421809] Key type ._llcrypt registered [ 2280.428533] Key type .llcrypt registered [ 2280.974282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2281.009221] alg: No test for adler32 (adler32-zlib) [ 2282.334421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2282.422890] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 2282.427435] LNet: Accept secure, port 988 [ 2294.932458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.132@tcp [ 2316.032325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2316.127630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2316.196181] LNet: Added LNI 192.168.206.32@tcp1 [8/256/0/180] [ 2320.414409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.132@tcp [ 2320.481647] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 2320.488475] LNet: 1 local NIs in recovery (showing 1): 192.168.206.32@tcp [ 2321.511876] LNet: There was an unexpected network error while writing to 192.168.206.32: rc = -22 [ 2321.535823] LNet: Skipped 2 previous similar messages [ 2322.486686] LNet: 46632:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2322.730420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2322.831243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2323.878536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.132@tcp [ 2323.940642] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 2323.943703] LNet: Skipped 1 previous similar message [ 2324.872851] LNet: 46700:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2325.213772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2325.327768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2326.602358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.132@tcp [ 2326.713594] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 2326.733796] LNet: Skipped 2 previous similar messages [ 2327.489739] LNet: 46768:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2327.822831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2327.964338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2329.420333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.132@tcp [ 2330.737908] LNet: 46837:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2330.966893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2331.040866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2332.249795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.132@tcp [ 2332.299971] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 2332.304972] LNet: Skipped 4 previous similar messages [ 2333.333398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2333.428875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2334.940137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.132@tcp [ 2335.691507] LNet: 46974:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2335.722589] LNet: 46974:0:(api-ni.c:337:recovery_interval_set()) Skipped 1 previous similar message [ 2336.019219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2336.145965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2346.436927] LNet: 47044:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2347.487507] LNet: Removed LNI 192.168.206.32@tcp [ 2347.503855] LNet: Removed LNI 192.168.206.32@tcp1 [ 2348.963664] Key type .llcrypt unregistered [ 2348.964981] Key type ._llcrypt unregistered [ 2390.975037] Key type ._llcrypt registered [ 2390.995657] Key type .llcrypt registered [ 2394.425195] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 04:12:49 (1743495169) [ 2396.924871] Key type .llcrypt unregistered [ 2396.926452] Key type ._llcrypt unregistered [ 2411.523249] Key type ._llcrypt registered [ 2411.528239] Key type .llcrypt registered [ 2412.219819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2412.270265] alg: No test for adler32 (adler32-zlib) [ 2413.507757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2413.577375] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 2413.581060] LNet: Accept secure, port 988 [ 2426.523916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.132@tcp [ 2427.864896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.132@tcp [ 2427.916381] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 2428.670510] LNet: 48254:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2428.928929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2429.007870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2429.704342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.132@tcp [ 2429.755606] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 2430.630441] LNet: 48316:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2430.929266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2431.048386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2432.040987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.132@tcp [ 2432.074785] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 2432.911613] LNet: 48379:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2433.121355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2433.264532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2434.117713] LNet: 48414:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2434.144243] LNet: Removed LNI 192.168.206.32@tcp [ 2435.522276] Key type .llcrypt unregistered [ 2435.523890] Key type ._llcrypt unregistered [ 2481.079701] Key type ._llcrypt registered [ 2481.080994] Key type .llcrypt registered [ 2484.553620] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 04:14:19 (1743495259) [ 2487.106400] Key type .llcrypt unregistered [ 2487.108873] Key type ._llcrypt unregistered [ 2503.212334] Key type ._llcrypt registered [ 2503.226158] Key type .llcrypt registered [ 2503.748874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2503.803782] alg: No test for adler32 (adler32-zlib) [ 2505.112370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2505.162199] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 2505.166178] LNet: Accept secure, port 988 [ 2518.875608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.132@tcp [ 2535.838494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2535.997486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2536.084145] LNet: Added LNI 192.168.206.32@tcp1 [8/256/0/180] [ 2539.939342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.132@tcp [ 2539.988299] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 2540.005255] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.132@tcp [ 2541.010959] LNet: 49832:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2541.027257] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 2541.030242] LNet: Skipped 3 previous similar messages [ 2541.237233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2541.347673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2541.469758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2542.596513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.132@tcp [ 2542.676106] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 2543.604967] LNet: 49904:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2543.758737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2543.840166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2543.986912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2544.910464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.132@tcp [ 2544.982647] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 2545.757115] LNet: 49974:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2545.949220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2546.137100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2546.319933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2555.572207] LNet: 50046:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2556.639238] LNet: Removed LNI 192.168.206.32@tcp [ 2556.660926] LNet: Removed LNI 192.168.206.32@tcp1 [ 2558.056049] Key type .llcrypt unregistered [ 2558.057829] Key type ._llcrypt unregistered [ 2599.775079] Key type ._llcrypt registered [ 2599.795961] Key type .llcrypt registered [ 2602.871675] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 04:16:18 (1743495378) [ 2604.195708] Key type .llcrypt unregistered [ 2604.199378] Key type ._llcrypt unregistered [ 2618.011605] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2618.515111] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2618.516738] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2618.518479] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2619.590986] Key type ._llcrypt registered [ 2619.592580] Key type .llcrypt registered [ 2619.947280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2619.982574] alg: No test for adler32 (adler32-zlib) [ 2621.414592] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 2621.426895] LNet: Accept secure, port 988 [ 2621.537113] LNet: 51116:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2621.557544] LNet: Removed LNI 192.168.206.32@tcp [ 2623.728537] Key type .llcrypt unregistered [ 2623.730210] Key type ._llcrypt unregistered [ 2641.789202] Key type ._llcrypt registered [ 2641.791050] Key type .llcrypt registered [ 2642.410995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2642.470485] alg: No test for adler32 (adler32-zlib) [ 2643.788208] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 2643.790590] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 2643.798732] LNet: Accept secure, port 988 [ 2643.891247] LNet: 51387:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2643.902204] LNet: Removed LNI 192.168.206.32@tcp [ 2643.914616] LNet: Removed LNI 10.1.2.3@tcp1 [ 2645.951886] Key type .llcrypt unregistered [ 2645.956949] Key type ._llcrypt unregistered [ 2661.420929] Key type ._llcrypt registered [ 2661.422597] Key type .llcrypt registered [ 2662.095661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2662.110705] alg: No test for adler32 (adler32-zlib) [ 2663.430557] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 2663.434859] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2663.442781] LNet: Accept secure, port 988 [ 2663.533663] LNet: 51658:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2663.540353] LNet: Removed LNI 192.168.206.32@tcp [ 2663.556350] LNet: Removed LNI 10.1.2.3@tcp [ 2665.887285] Key type .llcrypt unregistered [ 2665.888682] Key type ._llcrypt unregistered [ 2679.445723] Key type ._llcrypt registered [ 2679.451179] Key type .llcrypt registered [ 2679.871898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2679.909503] alg: No test for adler32 (adler32-zlib) [ 2681.249232] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 2681.251597] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2681.260324] LNet: Accept secure, port 988 [ 2681.346350] LNet: 51929:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2681.349874] LNet: Removed LNI 192.168.206.32@tcp [ 2681.365453] LNet: Removed LNI 10.1.2.3@tcp [ 2683.486291] Key type .llcrypt unregistered [ 2683.488128] Key type ._llcrypt unregistered [ 2703.950464] Key type ._llcrypt registered [ 2703.966705] Key type .llcrypt registered [ 2704.631306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2704.680843] alg: No test for adler32 (adler32-zlib) [ 2705.950231] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 2705.952719] LNetError: 52192:0:(config.c:1558:lnet_inet_select()) ksocklnd: failed to find interface test1pg [ 2705.956650] LNetError: Error -100 starting up LNI tcp [ 2705.970975] LNet: Removed LNI 192.168.206.32@tcp [ 2707.845373] Key type .llcrypt unregistered [ 2707.848650] Key type ._llcrypt unregistered [ 2736.358975] Key type ._llcrypt registered [ 2736.360573] Key type .llcrypt registered [ 2740.151245] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 04:18:35 (1743495515) [ 2742.775721] Key type .llcrypt unregistered [ 2742.787295] Key type ._llcrypt unregistered [ 2757.185260] Key type ._llcrypt registered [ 2757.186528] Key type .llcrypt registered [ 2757.666469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2757.727832] alg: No test for adler32 (adler32-zlib) [ 2758.935801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2759.012917] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 2759.033689] LNet: Accept secure, port 988 [ 2772.484418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.132@tcp [ 2773.559281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.132@tcp [ 2773.592867] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 2774.455182] LNet: 53404:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2774.614470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2774.701230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2775.331859] LNet: 53438:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2775.359353] LNet: Removed LNI 192.168.206.32@tcp [ 2776.621840] Key type .llcrypt unregistered [ 2776.634805] Key type ._llcrypt unregistered [ 2823.197207] Key type ._llcrypt registered [ 2823.198848] Key type .llcrypt registered [ 2824.014783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2824.058385] alg: No test for adler32 (adler32-zlib) [ 2825.295207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2825.357688] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 2825.361754] LNet: Accept secure, port 988 [ 2839.613417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.132@tcp [ 2858.271627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2858.418000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2858.498729] LNet: Added LNI 192.168.206.32@tcp1 [8/256/0/180] [ 2862.478971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.132@tcp [ 2862.532072] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 2863.265457] LNet: 1 local NIs in recovery (showing 1): 192.168.206.32@tcp [ 2863.267548] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.132@tcp [ 2863.270132] LNet: There was an unexpected network error while writing to 192.168.206.32: rc = -22 [ 2863.341535] LNet: 54268:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2863.580774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2863.647192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2869.869078] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 2869.872513] LNet: Skipped 1 previous similar message [ 2870.723788] LNet: 54337:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2871.775240] LNet: Removed LNI 192.168.206.32@tcp [ 2871.789530] LNet: Removed LNI 192.168.206.32@tcp1 [ 2873.141931] Key type .llcrypt unregistered [ 2873.145547] Key type ._llcrypt unregistered [ 2918.413552] Key type ._llcrypt registered [ 2918.415135] Key type .llcrypt registered [ 2922.087584] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 04:21:37 (1743495697) [ 2923.129973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2923.164840] alg: No test for adler32 (adler32-zlib) [ 2924.516191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2924.722467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2924.802563] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 2924.813699] LNet: Accept secure, port 988 [ 2927.116062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2927.182387] LNet: Added LNI 192.168.206.32@tcp1 [8/256/0/180] [ 2930.098642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.32@tcp [ 2930.311218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 2930.817256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.32@tcp [ 2930.891586] LNetError: 55430:0:(api-ni.c:4072:lnet_ni_set_healthv()) manually adding local NI 192.168.206.32@tcp to recovery [ 2931.808715] LNet: There was an unexpected network error while writing to 192.168.206.32: rc = -22 [ 2931.809719] LNet: 1 local NIs in recovery (showing 1): 192.168.206.32@tcp [ 2933.869353] LNet: There was an unexpected network error while writing to 192.168.206.32: rc = -22 [ 2935.369524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 2938.021042] LNet: There was an unexpected network error while writing to 192.168.206.32: rc = -22 [ 2938.620262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 2938.865893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2938.897959] LNet: 55526:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2938.907365] LNet: Removed LNI 192.168.206.32@tcp [ 2938.920535] LNet: Removed LNI 192.168.206.32@tcp1 [ 2938.973901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2939.110799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2939.166568] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 2939.173808] LNet: Accept secure, port 988 [ 2941.075776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2941.111384] LNet: Added LNI 192.168.206.32@tcp1 [8/256/0/180] [ 2943.653418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.32@tcp [ 2943.780308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2943.983481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 2944.328471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.32@tcp [ 2944.385049] LNetError: 55789:0:(api-ni.c:4072:lnet_ni_set_healthv()) manually adding local NI 192.168.206.32@tcp to recovery [ 2945.184539] LNet: There was an unexpected network error while writing to 192.168.206.32: rc = -22 [ 2948.858609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 2951.329769] LNet: There was an unexpected network error while writing to 192.168.206.32: rc = -22 [ 2951.333061] LNet: Skipped 1 previous similar message [ 2957.011522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 2957.144836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2976.299158] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 04:22:31 (1743495751) [ 2976.563467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2976.651828] LNet: 56495:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2977.695362] LNet: Removed LNI 192.168.206.32@tcp [ 2977.710719] LNet: Removed LNI 192.168.206.32@tcp1 [ 2977.822934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2977.970717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2978.019918] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 2978.026207] LNet: Accept secure, port 988 [ 2980.243693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2980.290299] LNet: Added LNI 192.168.206.32@tcp1 [8/256/0/180] [ 2982.787805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.32@tcp [ 2982.965561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 2983.267719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.32@tcp --health 0 [ 2983.677498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2984.032849] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.32@tcp [ 2984.038537] LNet: There was an unexpected network error while writing to 192.168.206.32: rc = -22 [ 2984.047910] LNet: Skipped 1 previous similar message [ 2999.126989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2999.586794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2999.689507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.32@tcp --health 500 [ 3000.063946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 3006.825970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 3007.060194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3007.128994] LNet: 56956:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3008.159359] LNet: Removed LNI 192.168.206.32@tcp [ 3008.187811] LNet: Removed LNI 192.168.206.32@tcp1 [ 3008.308230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3008.593906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3008.719344] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 3008.732846] LNet: Accept secure, port 988 [ 3011.145932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 3011.196889] LNet: Added LNI 192.168.206.32@tcp1 [8/256/0/180] [ 3013.820032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.32@tcp [ 3013.976290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3014.153671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 3014.526150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.32@tcp --health 0 [ 3015.584118] LNet: There was an unexpected network error while writing to 192.168.206.32: rc = -22 [ 3015.592413] LNet: Skipped 3 previous similar messages [ 3016.612607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 3026.575531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 3026.702856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3044.262462] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.32@tcp [ 3044.609757] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 04:23:40 (1743495820) [ 3046.088181] LNet: 57972:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3047.135378] LNet: Removed LNI 192.168.206.32@tcp [ 3047.160788] LNet: Removed LNI 192.168.206.32@tcp1 [ 3049.285754] Key type .llcrypt unregistered [ 3049.288741] Key type ._llcrypt unregistered [ 3067.870937] Key type ._llcrypt registered [ 3067.876137] Key type .llcrypt registered [ 3068.454808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3068.491302] alg: No test for adler32 (adler32-zlib) [ 3069.940643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 3070.036850] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 3070.044264] LNet: Accept secure, port 988 [ 3089.720165] Lustre: DEBUG MARKER: Initial discovery [ 3090.580160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.132@tcp [ 3097.234840] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 3097.397765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.132@tcp [ 3097.469214] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 3101.883962] Lustre: DEBUG MARKER: Force oleg632-client.virtnet to discover 192.168.206.132@tcp (in background) [ 3102.065307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.132@tcp [ 3105.208962] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 3105.547232] Lustre: DEBUG MARKER: Force oleg632-server to discover 192.168.206.32@tcp [ 3111.140841] LNet: There was an unexpected network error while writing to 192.168.206.132: rc = -22 [ 3111.697108] Lustre: DEBUG MARKER: Wait for 58721 [ 3115.346967] Lustre: DEBUG MARKER: Finished wait on 58721 [ 3116.517751] LNet: 59177:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3116.548362] LNet: Removed LNI 192.168.206.32@tcp [ 3117.934099] Key type .llcrypt unregistered [ 3117.935538] Key type ._llcrypt unregistered [ 3163.705893] Key type ._llcrypt registered [ 3163.707552] Key type .llcrypt registered [ 3167.685881] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 04:25:42 (1743495942) [ 3169.421177] Key type .llcrypt unregistered [ 3169.423298] Key type ._llcrypt unregistered [ 3186.176224] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 3187.095171] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 3187.111606] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 3187.122756] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 3187.970898] Key type ._llcrypt registered [ 3187.985896] Key type .llcrypt registered [ 3188.676907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3188.734591] alg: No test for adler32 (adler32-zlib) [ 3190.006415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3190.300834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3190.407908] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 3190.411623] LNet: Accept secure, port 988 [ 3192.692376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 3192.748844] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 3218.301351] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 04:26:33 (1743495993) [ 3218.957778] LNet: 61228:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3218.965057] LNet: Removed LNI 192.168.206.32@tcp [ 3218.982505] LNet: Removed LNI 10.1.2.3@tcp [ 3221.012260] Key type .llcrypt unregistered [ 3221.014445] Key type ._llcrypt unregistered [ 3237.774557] Key type ._llcrypt registered [ 3237.779820] Key type .llcrypt registered [ 3238.298192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3238.347443] alg: No test for adler32 (adler32-zlib) [ 3239.583345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3239.775604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3239.837661] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 3239.843678] LNet: Accept secure, port 988 [ 3241.646043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 3241.690972] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 3244.393424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.206.32@tcp [ 3245.535735] LNetError: Unexpected error -22 connecting to 192.168.206.32@tcp at host 192.168.206.32:988 [ 3260.776251] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 04:27:15 (1743496035) [ 3261.098534] LNet: 62340:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3262.176516] LNet: Removed LNI 192.168.206.32@tcp [ 3262.198349] LNet: Removed LNI 10.1.2.3@tcp [ 3264.309078] Key type .llcrypt unregistered [ 3264.316492] Key type ._llcrypt unregistered [ 3279.794590] Key type ._llcrypt registered [ 3279.796142] Key type .llcrypt registered [ 3280.377342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3280.403029] alg: No test for adler32 (adler32-zlib) [ 3281.640288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3281.773659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 3281.819566] LNet: 62612:0:(lib-socket.c:568:lnet_inet_enumerate()) lnet: Ignoring interface test1pg: it's down [ 3281.829647] LNet: Added LNI 192.168.206.32@tcp1 [8/256/0/180] [ 3281.838610] LNet: Accept secure, port 988 [ 3283.884363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3283.929698] LNet: 62732:0:(lib-socket.c:568:lnet_inet_enumerate()) lnet: Ignoring interface test1pg: it's down [ 3283.945398] LNet: Added LNI 192.168.206.32@tcp2 [8/256/0/180] [ 3286.633601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.206.32@tcp1 --nid 192.168.206.32@tcp2 [ 3299.124344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.32@tcp1 192.168.206.32@tcp2 [ 3299.189670] LNetError: 63176:0:(lib-move.c:2225:lnet_handle_find_routed_path()) Unable to send message from 192.168.206.32@tcp1 to 192.168.206.32@tcp2 - Route table may be misconfigured [ 3299.258118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.32@tcp2 192.168.206.32@tcp1 [ 3316.179995] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 04:28:11 (1743496091) [ 3316.478485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3316.557250] LNet: 63745:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3317.599351] LNet: Removed LNI 192.168.206.32@tcp1 [ 3318.646040] LNet: Removed LNI 192.168.206.32@tcp2 [ 3318.748821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3318.974595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3319.081470] LNet: 63758:0:(lib-socket.c:568:lnet_inet_enumerate()) lnet: Ignoring interface test1pg: it's down [ 3319.099377] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 3319.112270] LNet: Accept secure, port 988 [ 3321.343425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 3321.416486] LNet: 63877:0:(lib-socket.c:568:lnet_inet_enumerate()) lnet: Ignoring interface test1pg: it's down [ 3321.422220] LNet: Added LNI 192.168.206.32@tcp1 [8/256/0/180] [ 3323.900811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.32@tcp [ 3323.976199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.32@tcp 192.168.206.32@tcp [ 3324.048979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.32@tcp1 192.168.206.32@tcp1 [ 3324.287176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 3324.353913] LNet: There was an unexpected network error while writing to 192.168.206.32: rc = -22 [ 3324.654712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 3324.962365] LNet: 1 local NIs in recovery (showing 1): 192.168.206.32@tcp [ 3324.990430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 3342.496895] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 04:28:37 (1743496117) [ 3342.770426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3342.795062] LNet: 64598:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3343.839235] LNet: Removed LNI 192.168.206.32@tcp [ 3344.944201] LNet: Removed LNI 192.168.206.32@tcp1 [ 3345.029680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3345.385785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 3346.064882] LNet: 64642:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3347.311674] Key type .llcrypt unregistered [ 3347.314110] Key type ._llcrypt unregistered [ 3377.444113] Key type ._llcrypt registered [ 3377.446410] Key type .llcrypt registered [ 3380.576170] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 04:29:16 (1743496156) [ 3381.548875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3381.597725] alg: No test for adler32 (adler32-zlib) [ 3382.837259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3386.166555] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 3389.846169] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 04:29:25 (1743496165) [ 3390.426261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3390.477860] LNet: 65743:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3390.572767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3390.721377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3390.789165] LNet: 65750:0:(lib-socket.c:568:lnet_inet_enumerate()) lnet: Ignoring interface test1pg: it's down [ 3390.799773] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 3390.808246] LNet: Accept secure, port 988 [ 3393.199304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 3393.244180] LNet: 65868:0:(lib-socket.c:568:lnet_inet_enumerate()) lnet: Ignoring interface test1pg: it's down [ 3393.248078] LNet: Added LNI 192.168.206.32@tcp1 [8/256/0/180] [ 3395.422423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 3395.547698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp1 [ 3395.673743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.32@tcp1 [ 3412.378973] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 04:29:47 (1743496187) [ 3417.161355] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg632-server [ 3421.872594] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 04:29:57 (1743496197) [ 3427.881457] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg632-server [ 3433.141300] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 04:30:07 (1743496207) [ 3439.112559] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg632-server [ 3443.268881] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 04:30:18 (1743496218) [ 3449.034861] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg632-server [ 3454.429633] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 04:30:29 (1743496229) [ 3460.452398] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg632-server [ 3464.855987] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 04:30:40 (1743496240) [ 3469.945590] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg632-server [ 3474.672864] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 04:30:50 (1743496250) [ 3479.407384] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg632-server [ 3483.344526] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 04:30:59 (1743496259) [ 3488.067896] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg632-server [ 3492.052430] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 04:31:07 (1743496267) [ 3492.369435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3492.435816] LNet: 69283:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3493.471285] LNet: Removed LNI 192.168.206.32@tcp [ 3494.501977] LNet: Removed LNI 192.168.206.32@tcp1 [ 3494.589907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3494.775980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3494.840358] LNet: 69296:0:(lib-socket.c:568:lnet_inet_enumerate()) lnet: Ignoring interface test1pg: it's down [ 3494.850363] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 3494.855420] LNet: Accept secure, port 988 [ 3496.924827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 3497.262661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 3498.349235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3498.434549] LNet: 69433:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3499.507582] LNet: Removed LNI 192.168.206.32@tcp [ 3499.609915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3499.838225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 3499.916063] LNet: 69440:0:(lib-socket.c:568:lnet_inet_enumerate()) lnet: Ignoring interface test1pg: it's down [ 3499.923448] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 3499.926887] LNet: Accept secure, port 988 [ 3502.554402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 3503.273925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3503.327988] LNet: 69576:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3504.361709] LNet: Removed LNI 192.168.206.32@tcp [ 3504.499952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3504.827336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3504.916747] LNet: 69583:0:(lib-socket.c:568:lnet_inet_enumerate()) lnet: Ignoring interface test1pg: it's down [ 3504.926556] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 3504.930339] LNet: Accept secure, port 988 [ 3507.509452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 3507.787111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 3508.305174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3508.340542] LNet: 69720:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3509.419223] LNet: Removed LNI 192.168.206.32@tcp [ 3509.537726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3509.702117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 3509.757610] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 3509.761142] LNet: Accept secure, port 988 [ 3512.186371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 3512.798773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3513.968286] LNet: Removed LNI 192.168.206.32@tcp [ 3514.048158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3514.305745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3514.387393] LNet: 69870:0:(lib-socket.c:568:lnet_inet_enumerate()) lnet: Ignoring interface test1pg: it's down [ 3514.396801] LNet: 69870:0:(lib-socket.c:568:lnet_inet_enumerate()) Skipped 1 previous similar message [ 3514.431456] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 3514.435673] LNet: Accept secure, port 988 [ 3516.520650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 3516.887223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 3518.847556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3518.919747] LNet: 70017:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3518.928408] LNet: 70017:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 3519.973787] LNet: Removed LNI 192.168.206.32@tcp [ 3520.090782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3520.308533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 3520.418603] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 3520.428642] LNet: Accept secure, port 988 [ 3523.054554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 3525.043867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3526.126594] LNet: Removed LNI 192.168.206.32@tcp [ 3526.233314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3526.458638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3526.563116] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 3526.569474] LNet: Accept secure, port 988 [ 3529.106627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 3529.380218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 3529.979392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3531.050223] LNet: Removed LNI 192.168.206.32@tcp [ 3531.163632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3531.360659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 3531.414614] LNet: 70321:0:(lib-socket.c:568:lnet_inet_enumerate()) lnet: Ignoring interface test1pg: it's down [ 3531.420635] LNet: 70321:0:(lib-socket.c:568:lnet_inet_enumerate()) Skipped 2 previous similar messages [ 3531.441051] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 3531.448638] LNet: Accept secure, port 988 [ 3534.371913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 3534.981369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3535.082194] LNet: 70456:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3535.085125] LNet: 70456:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 3536.165632] LNet: Removed LNI 192.168.206.32@tcp [ 3536.285544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3536.543504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3536.606230] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 3536.610966] LNet: Accept secure, port 988 [ 3538.923914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 3539.203925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 3539.855873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3540.965214] LNet: Removed LNI 192.168.206.32@tcp [ 3541.045833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3541.288227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 3541.362248] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 3541.370021] LNet: Accept secure, port 988 [ 3544.072647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 3547.394049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3548.461762] LNet: Removed LNI 192.168.206.32@tcp [ 3548.582791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3548.730808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3548.793784] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 3548.797847] LNet: Accept secure, port 988 [ 3551.172508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 3551.406929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 3556.137299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3557.287458] LNet: Removed LNI 192.168.206.32@tcp [ 3557.369619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3557.486973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 3557.537500] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 3557.544307] LNet: Accept secure, port 988 [ 3559.776634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 3561.913566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3562.983094] LNet: Removed LNI 192.168.206.32@tcp [ 3563.066902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3563.197837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3563.240697] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 3563.247824] LNet: Accept secure, port 988 [ 3565.249057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 3565.394354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3565.478610] LNet: Removed LNI 192.168.206.32@tcp [ 3565.556365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3565.731307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3565.772761] LNet: 71223:0:(lib-socket.c:568:lnet_inet_enumerate()) lnet: Ignoring interface test1pg: it's down [ 3565.777700] LNet: 71223:0:(lib-socket.c:568:lnet_inet_enumerate()) Skipped 5 previous similar messages [ 3565.784744] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 3565.789156] LNet: Accept secure, port 988 [ 3567.590070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 3584.476892] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 04:32:40 (1743496360) [ 3585.001823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3585.173303] LNet: 71917:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3585.176445] LNet: 71917:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 3585.202643] LNet: Removed LNI 192.168.206.32@tcp [ 3585.292468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3585.474262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 3585.533857] LNet: Added LNI 192.168.206.32@tcp231 [8/256/0/180] [ 3585.542708] LNet: Accept secure, port 988 [ 3588.556288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3588.657245] LNet: Removed LNI 192.168.206.32@tcp231 [ 3588.738320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3588.980804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 3589.042891] LNet: Added LNI 192.168.206.32@tcp231 [8/256/0/180] [ 3589.046589] LNet: Accept secure, port 988 [ 3589.050556] LNet: 72062:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3589.845260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 3589.951369] LNet: Removed LNI 192.168.206.32@tcp231 [ 3590.025957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 3590.080528] LNet: Added LNI 192.168.206.32@tcp231 [8/256/0/180] [ 3592.832385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3592.920105] LNet: Removed LNI 192.168.206.32@tcp231 [ 3592.961428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3593.210607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 3593.299460] LNet: Added LNI 192.168.206.32@tcp231 [8/256/0/0] [ 3593.304659] LNet: Accept secure, port 988 [ 3593.310537] LNet: 72212:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3593.925237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 3594.008296] LNet: Removed LNI 192.168.206.32@tcp231 [ 3594.074568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 3594.115266] LNet: Added LNI 192.168.206.32@tcp231 [8/256/0/0] [ 3596.965706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3597.042864] LNet: Removed LNI 192.168.206.32@tcp231 [ 3597.131638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3597.391985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 3597.458040] LNet: Added LNI 192.168.206.32@tcp231 [8/256/0/60] [ 3597.467295] LNet: Accept secure, port 988 [ 3597.480923] LNet: 72360:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3598.095911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 3598.196189] LNet: Removed LNI 192.168.206.32@tcp231 [ 3598.266202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 3598.340403] LNet: Added LNI 192.168.206.32@tcp231 [8/256/0/60] [ 3601.598864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3601.677756] LNet: Removed LNI 192.168.206.32@tcp231 [ 3601.768752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3601.980915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 3602.050049] LNet: Added LNI 192.168.206.32@tcp231 [8/256/0/180] [ 3602.057183] LNet: Accept secure, port 988 [ 3602.061825] LNet: 72509:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3602.654478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 3602.734551] LNet: Removed LNI 192.168.206.32@tcp231 [ 3602.814587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 3602.937388] LNet: Added LNI 192.168.206.32@tcp231 [8/256/0/180] [ 3628.225086] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 04:33:22 (1743496402) [ 3628.522618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3628.681055] LNet: Removed LNI 192.168.206.32@tcp231 [ 3628.877127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3629.351758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 3629.432202] LNet: Added LNI 192.168.206.32@tcp232 [8/256/0/180] [ 3629.448202] LNet: Accept secure, port 988 [ 3633.078593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 3633.173897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 3633.223369] LNet: Removed LNI 192.168.206.32@tcp232 [ 3633.327802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 3633.428604] LNet: 73353:0:(lib-socket.c:568:lnet_inet_enumerate()) lnet: Ignoring interface test1pg: it's down [ 3633.440901] LNet: 73353:0:(lib-socket.c:568:lnet_inet_enumerate()) Skipped 10 previous similar messages [ 3633.456665] LNet: Added LNI 192.168.206.32@tcp232 [8/256/0/180] [ 3635.911271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 3636.009747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 3636.115270] LNet: Removed LNI 192.168.206.32@tcp232 [ 3636.174957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 3636.321131] LNet: Added LNI 192.168.206.32@tcp232 [8/256/0/180] [ 3639.570752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 3639.757052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 3639.868810] LNet: Removed LNI 192.168.206.32@tcp232 [ 3639.950346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 3640.032961] LNet: Added LNI 192.168.206.32@tcp232 [8/256/0/180] [ 3642.732795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 3642.821765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 3642.882812] LNet: Removed LNI 192.168.206.32@tcp232 [ 3642.919159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 3660.742919] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 04:33:56 (1743496436) [ 3661.614444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3661.689195] LNet: 74298:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3661.691734] LNet: 74298:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 3661.776376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3661.898592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3661.962517] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 3661.976206] LNet: Accept secure, port 988 [ 3679.438428] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 04:34:15 (1743496455) [ 3683.302475] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 3687.530411] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 04:34:23 (1743496463) [ 3689.001182] LNet: Removed LNI 192.168.206.32@tcp [ 3691.231514] Key type .llcrypt unregistered [ 3691.237485] Key type ._llcrypt unregistered [ 3710.064857] Key type ._llcrypt registered [ 3710.066378] Key type .llcrypt registered [ 3710.645436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3710.673395] alg: No test for adler32 (adler32-zlib) [ 3711.940409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 3712.053115] LNet: 75593:0:(lib-socket.c:568:lnet_inet_enumerate()) lnet: Ignoring interface test1pg: it's down [ 3712.101920] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 3712.105750] LNet: Accept secure, port 988 [ 3727.145490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.132@tcp [ 3738.664359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.206.132@tcp [ 3739.476385] LNet: 75753:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3739.489941] LNet: Removed LNI 192.168.206.32@tcp [ 3740.541665] Key type .llcrypt unregistered [ 3740.543254] Key type ._llcrypt unregistered [ 3769.799265] Key type ._llcrypt registered [ 3769.805352] Key type .llcrypt registered [ 3773.634935] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 3777.577738] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 3781.583339] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 04:35:57 (1743496557) [ 3782.714741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3782.760920] alg: No test for adler32 (adler32-zlib) [ 3783.985800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3784.251649] LNet: 76842:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3786.280830] Key type .llcrypt unregistered [ 3786.287800] Key type ._llcrypt unregistered [ 3802.650324] Key type ._llcrypt registered [ 3802.651556] Key type .llcrypt registered [ 3803.086657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3803.109807] alg: No test for adler32 (adler32-zlib) [ 3804.395464] LNet: 77095:0:(lib-socket.c:568:lnet_inet_enumerate()) lnet: Ignoring interface test1pg: it's down [ 3804.407574] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 3804.416274] LNet: 77095:0:(router.c:737:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 3804.438634] LNet: Accept secure, port 988 [ 3804.460768] LNetError: 77096:0:(lib-lnet.h:1313:lnet_set_route_aliveness()) route to o2ib through 192.168.206.33@tcp has gone from up to down [ 3804.495666] LNetError: 77107:0:(router.c:400:lnet_router_discovery_ping_reply()) Peer 192.168.206.34@tcp is being used as a gateway but routing feature is not turned on [ 3804.914815] LNetError: 77126:0:(router.c:334:lnet_is_route_alive()) peer 192.168.206.34@tcp is being used as a gateway but routing feature is not turned on [ 3805.876835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3805.958655] LNet: 77151:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3807.021564] LNet: Removed LNI 192.168.206.32@tcp [ 3807.094854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 3807.164927] LNet: 77154:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3822.601569] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 04:36:38 (1743496598) [ 3827.139721] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg632-server [ 3831.575518] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 04:36:46 (1743496606) [ 3838.107681] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 3838.798415] LNet: 78168:0:(lib-socket.c:568:lnet_inet_enumerate()) lnet: Ignoring interface test1pg: it's down [ 3838.803971] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 3838.806377] LNet: 78168:0:(router.c:737:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 3838.816407] LNet: 78168:0:(router.c:737:lnet_add_route()) Skipped 4 previous similar messages [ 3838.823684] LNet: Accept secure, port 988 [ 3838.883306] LNetError: 78183:0:(router.c:400:lnet_router_discovery_ping_reply()) Peer 192.168.206.34@tcp is being used as a gateway but routing feature is not turned on [ 3838.890763] LNetError: 78186:0:(lib-lnet.h:1313:lnet_set_route_aliveness()) route to o2ib through 192.168.206.36@tcp has gone from up to down [ 3838.900050] LNetError: 78183:0:(router.c:400:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 3838.909695] LNetError: 78186:0:(lib-lnet.h:1313:lnet_set_route_aliveness()) Skipped 5 previous similar messages [ 3840.839423] Key type lgssc registered [ 3844.571645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3895.391178] Key type lgssc unregistered [ 3896.108191] LNet: 79109:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3897.193529] LNet: Removed LNI 192.168.206.32@tcp [ 3899.025669] Key type .llcrypt unregistered [ 3899.031949] Key type ._llcrypt unregistered [ 3942.033230] Key type ._llcrypt registered [ 3942.035292] Key type .llcrypt registered [ 3945.801731] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 04:38:40 (1743496720) [ 3947.897935] Key type .llcrypt unregistered [ 3947.899411] Key type ._llcrypt unregistered [ 3966.304471] Key type ._llcrypt registered [ 3966.313996] Key type .llcrypt registered [ 3966.820796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3966.850773] alg: No test for adler32 (adler32-zlib) [ 3986.311146] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 04:39:21 (1743496761) [ 3986.767912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3986.829352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3986.937293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3986.981310] LNet: 80717:0:(lib-socket.c:568:lnet_inet_enumerate()) lnet: Ignoring interface test1pg: it's down [ 3986.999053] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 3987.004317] LNet: Accept secure, port 988 [ 3989.474983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3989.580579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 3989.675876] LNet: Removed LNI 192.168.206.32@tcp [ 3990.512074] LNet: 80865:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3992.077588] Key type .llcrypt unregistered [ 3992.079416] Key type ._llcrypt unregistered [ 4027.797881] Key type ._llcrypt registered [ 4027.852050] Key type .llcrypt registered [ 4031.552403] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 04:40:06 (1743496806) [ 4032.594443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4032.636593] alg: No test for adler32 (adler32-zlib) [ 4033.946600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 4034.156656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 4034.202674] LNet: 81675:0:(lib-socket.c:568:lnet_inet_enumerate()) lnet: Ignoring interface test1pg: it's down [ 4034.217833] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 4034.223353] LNet: Accept secure, port 988 [ 4036.358713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 4036.434859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.206.32@tcp [ 4036.473486] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 4036.475872] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 4036.478584] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 4036, 0, 0 [ 4036.482988] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 4036.485565] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 4036.489136] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 4036.492866] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 4036, 0, 0 [ 4036.497157] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 4036.503394] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 4036.505894] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 4036.508029] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 4036, 0, 4085 [ 4036.512704] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 4036.515521] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 4036.526154] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 4036.534137] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 4036, 0, 4085 [ 4036.538036] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 4052.937669] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 04:40:28 (1743496828) [ 4054.291776] LNet: 82414:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4055.333976] LNet: Removed LNI 192.168.206.32@tcp [ 4057.518267] Key type .llcrypt unregistered [ 4057.519879] Key type ._llcrypt unregistered [ 4074.996216] Key type ._llcrypt registered [ 4074.997562] Key type .llcrypt registered [ 4075.567584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 4075.596685] alg: No test for adler32 (adler32-zlib) [ 4076.832555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 4076.900155] LNet: 82673:0:(lib-socket.c:568:lnet_inet_enumerate()) lnet: Ignoring interface test1pg: it's down [ 4076.916973] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 4076.922528] LNet: Accept secure, port 988 [ 4091.600715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.132@tcp [ 4107.715136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 4107.818525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 4107.858241] LNet: 82842:0:(lib-socket.c:568:lnet_inet_enumerate()) lnet: Ignoring interface test1pg: it's down [ 4107.862438] LNet: Added LNI 192.168.206.32@tcp1 [8/256/0/180] [ 4110.999926] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 4111.007074] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 4111.181547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 4111.253763] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 4143.140248] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 04:41:57 (1743496917) [ 4143.431249] LNet: 83885:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4144.482744] LNet: Removed LNI 192.168.206.32@tcp [ 4145.503231] LNet: Removed LNI 192.168.206.32@tcp1 [ 4145.524138] LNet: Removed LNI 10.1.2.3@tcp99 [ 4147.301100] Key type .llcrypt unregistered [ 4147.316528] Key type ._llcrypt unregistered [ 4165.746397] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 4165.767823] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 4166.642761] Key type ._llcrypt registered [ 4166.644319] Key type .llcrypt registered [ 4167.307090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 4167.350597] alg: No test for adler32 (adler32-zlib) [ 4168.690909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 4168.952073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 4169.017900] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 4169.036117] LNet: Accept secure, port 988 [ 4171.183051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 4171.220100] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 4178.101210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 4178.193626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 4178.307478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 4178.356563] LNetError: 84607:0:(peer.c:2185:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 4178.414820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 4178.553989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.32@tcp [ 4179.131345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 4179.353135] LNet: 84628:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4180.385906] LNet: Removed LNI 192.168.206.32@tcp [ 4180.418351] LNet: Removed LNI 10.1.2.3@tcp [ 4182.646638] Key type .llcrypt unregistered [ 4182.653138] Key type ._llcrypt unregistered [ 4211.816442] Key type ._llcrypt registered [ 4211.817632] Key type .llcrypt registered [ 4215.003746] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 04:43:10 (1743496990) [ 4216.013496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4216.071902] alg: No test for adler32 (adler32-zlib) [ 4217.320417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 4217.435203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 4217.480734] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 4217.485378] LNet: Accept secure, port 988 [ 4219.831277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.32@tcp [ 4220.026246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg632-client@tcp [ 4236.221919] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 04:43:31 (1743497011) [ 4236.591474] LNet: 86158:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4237.673255] LNet: Removed LNI 192.168.206.32@tcp [ 4239.957074] Key type .llcrypt unregistered [ 4239.958760] Key type ._llcrypt unregistered [ 4258.069838] Key type ._llcrypt registered [ 4258.073841] Key type .llcrypt registered [ 4258.550602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4258.617031] alg: No test for adler32 (adler32-zlib) [ 4259.839159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 4259.921824] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 4259.929177] LNet: Accept secure, port 988 [ 4260.038955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 4274.960751] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 04:44:10 (1743497050) [ 4278.479649] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 4283.356500] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 04:44:18 (1743497058) [ 4287.160342] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 4291.974779] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 04:44:27 (1743497067) [ 4292.596816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 4292.643458] LNet: 87586:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4292.664323] LNet: Removed LNI 192.168.206.32@tcp [ 4292.752848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 4292.943802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 4293.037287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 4293.793426] LNet: 87621:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4295.162141] Key type .llcrypt unregistered [ 4295.163827] Key type ._llcrypt unregistered [ 4322.887545] Key type ._llcrypt registered [ 4322.891346] Key type .llcrypt registered [ 4326.539768] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 04:45:01 (1743497101) [ 4327.661992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4327.701129] alg: No test for adler32 (adler32-zlib) [ 4328.946299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 4329.126569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 4329.191791] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 4329.196389] LNet: Accept secure, port 988 [ 4331.437495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 4331.688827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.32@tcp [ 4346.359210] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 04:45:22 (1743497122) [ 4346.923524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 4347.019989] LNet: 89140:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4348.071234] LNet: Removed LNI 192.168.206.32@tcp [ 4348.170559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 4348.343060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 4348.451045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 4363.162445] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 04:45:39 (1743497139) [ 4363.849353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 4363.937281] LNet: 89724:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4363.976427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 4364.495301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 4364.540352] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 4364.547245] LNet: Accept secure, port 988 [ 4366.309229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 4366.357551] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 4369.077754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.32@tcp [ 4369.617524] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 4369.620706] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 4370.350392] LNetError: 89749:0:(lib-move.c:4908:lnet_parse()) 192.168.206.32@tcp, src 192.168.206.32@tcp: Dropping PUT (error -108 looking up sender) [ 4371.361845] LNet: 90022:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4372.383693] LNet: Removed LNI 192.168.206.32@tcp [ 4372.397920] LNet: Removed LNI 10.1.2.3@tcp [ 4373.540043] Key type .llcrypt unregistered [ 4373.542336] Key type ._llcrypt unregistered [ 4405.359604] Key type ._llcrypt registered [ 4405.364609] Key type .llcrypt registered [ 4408.236646] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 04:46:23 (1743497183) [ 4409.649943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4409.696343] alg: No test for adler32 (adler32-zlib) [ 4411.132756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 4412.242465] LNet: 90876:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4414.441119] Key type .llcrypt unregistered [ 4414.444445] Key type ._llcrypt unregistered [ 4431.723402] Key type ._llcrypt registered [ 4431.724638] Key type .llcrypt registered [ 4432.271238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 4432.291449] alg: No test for adler32 (adler32-zlib) [ 4433.485153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 4433.585242] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 4433.592158] LNet: Accept secure, port 988 [ 4447.001559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.132@tcp [ 4448.520399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.132@tcp [ 4448.991303] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.132@tcp [ 4449.325637] LNet: 91283:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 4449.535990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 4449.590169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 4450.645721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 4451.418679] LNet: 91356:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 4451.609772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 4451.666772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 4452.590553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.132@tcp [ 4453.420505] LNet: 91430:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 4453.584700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 4453.645350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 4454.629793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 4455.541976] LNet: 91503:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 4455.946875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 4456.074891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 4457.108828] LNet: 91544:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4458.155144] LNet: Removed LNI 192.168.206.32@tcp [ 4459.357527] Key type .llcrypt unregistered [ 4459.360252] Key type ._llcrypt unregistered [ 4501.297738] Key type ._llcrypt registered [ 4501.323206] Key type .llcrypt registered [ 4504.502754] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 04:47:59 (1743497279) [ 4505.774918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4505.806390] alg: No test for adler32 (adler32-zlib) [ 4507.025173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 4508.230077] LNet: 92433:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4510.019090] Key type .llcrypt unregistered [ 4510.020664] Key type ._llcrypt unregistered [ 4526.873705] Key type ._llcrypt registered [ 4526.876676] Key type .llcrypt registered [ 4527.438414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 4527.479930] alg: No test for adler32 (adler32-zlib) [ 4528.812887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 4528.901261] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 4528.906913] LNet: Accept secure, port 988 [ 4541.701496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.132@tcp [ 4558.330545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 4558.429884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 4558.501965] LNet: Added LNI 192.168.206.32@tcp1 [8/256/0/180] [ 4562.107736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.132@tcp [ 4562.720241] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.132@tcp [ 4562.885078] LNet: 93041:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 4563.072744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 4563.177496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 4564.195991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 4564.927119] LNet: 93114:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 4565.129623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 4565.268876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 4566.624932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.132@tcp [ 4567.622751] LNet: 93188:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 4567.833897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 4567.912531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 4568.938509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 4569.683500] LNet: 93261:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 4569.970712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 4570.084494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 4579.358416] LNet: 93337:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4580.383325] LNet: Removed LNI 192.168.206.32@tcp [ 4580.409227] LNet: Removed LNI 192.168.206.32@tcp1 [ 4581.934410] Key type .llcrypt unregistered [ 4581.936181] Key type ._llcrypt unregistered [ 4619.089758] Key type ._llcrypt registered [ 4619.091401] Key type .llcrypt registered [ 4621.948745] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 3939 sec ========== 04:49:57 (1743497397) [ 4624.049698] Key type .llcrypt unregistered [ 4624.051509] Key type ._llcrypt unregistered [ 4652.090935] Key type ._llcrypt registered [ 4652.092527] Key type .llcrypt registered [ 4653.732952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 4653.767931] alg: No test for adler32 (adler32-zlib) [ 4655.454461] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 4656.045300] LNet: Added LNI 192.168.206.32@tcp [8/256/0/180] [ 4656.052802] LNet: Accept secure, port 988 [ 4658.129119] Key type lgssc registered [ 4661.152641] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4773.965693] Lustre: Mounted lustre-client [ 4783.904588] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4799.455449] Lustre: lustre-OST0000-osc-ffff9984f1c21000: disconnect after 22s idle [ 4803.023273] Lustre: Unmounted lustre-client [ 4848.042619] Key type lgssc unregistered [ 4848.786346] LNet: 96898:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4849.831258] LNet: Removed LNI 192.168.206.32@tcp [ 4851.800592] Key type .llcrypt unregistered [ 4851.803643] Key type ._llcrypt unregistered