[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 475836300 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002365] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008016] pid_max: default: 32768 minimum: 301 [ 0.009138] LSM: Security Framework initializing [ 0.010110] Yama: becoming mindful. [ 0.011043] SELinux: Initializing. [ 0.012072] *** VALIDATE selinux *** [ 0.021180] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025848] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027045] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028112] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029148] *** VALIDATE tmpfs *** [ 0.031406] *** VALIDATE proc *** [ 0.032249] *** VALIDATE cgroup *** [ 0.033012] *** VALIDATE cgroup2 *** [ 0.034272] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035210] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037029] Spectre V2 : User space: Vulnerable [ 0.038010] Speculative Store Bypass: Vulnerable [ 0.041250] debug: unmapping init [mem 0xffffffffaa059000-0xffffffffaa060fff] [ 0.043804] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044682] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045027] ... version: 2 [ 0.046018] ... bit width: 48 [ 0.047016] ... generic registers: 4 [ 0.048015] ... value mask: 0000ffffffffffff [ 0.049018] ... max period: 00007fffffffffff [ 0.050015] ... fixed-purpose events: 3 [ 0.051015] ... event mask: 000000070000000f [ 0.052327] rcu: Hierarchical SRCU implementation. [ 0.054581] smp: Bringing up secondary CPUs ... [ 0.055746] x86: Booting SMP configuration: [ 0.056030] .... node #0, CPUs: #1 #2 #3 [ 0.059172] smp: Brought up 1 node, 4 CPUs [ 0.061017] smpboot: Max logical packages: 1 [ 0.062015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.143023] node 0 deferred pages initialised in 78ms [ 0.146128] devtmpfs: initialized [ 0.147158] x86/mm: Memory block size: 128MB [ 0.149814] gcov: version magic: 0x41383552 [ 0.151334] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.152087] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.153285] pinctrl core: initialized pinctrl subsystem [ 0.154243] [ 0.154775] ************************************************************* [ 0.155016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156011] ** ** [ 0.157010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.158011] ** ** [ 0.159008] ** This means that this kernel is built to expose internal ** [ 0.160010] ** IOMMU data structures, which may compromise security on ** [ 0.161013] ** your system. ** [ 0.162011] ** ** [ 0.163016] ** If you see this message and you are not debugging the ** [ 0.164012] ** kernel, report this immediately to your vendor! ** [ 0.165017] ** ** [ 0.166011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.167014] ************************************************************* [ 0.168721] NET: Registered protocol family 16 [ 0.169462] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.170047] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.171042] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.172385] cpuidle: using governor menu [ 0.173719] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.175320] PCI: Using configuration type 1 for base access [ 0.176089] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.183097] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.184024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.186059] cryptd: max_cpu_qlen set to 1000 [ 0.189061] ACPI: Added _OSI(Module Device) [ 0.190032] ACPI: Added _OSI(Processor Device) [ 0.191000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.192019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.197671] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.203544] ACPI: Interpreter enabled [ 0.204054] ACPI: PM: (supports S0 S3 S4 S5) [ 0.205015] ACPI: Using IOAPIC for interrupt routing [ 0.207131] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.209368] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.220346] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.223044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.225020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.228100] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.233381] acpiphp: Slot [2] registered [ 0.235128] acpiphp: Slot [5] registered [ 0.236160] acpiphp: Slot [6] registered [ 0.237131] acpiphp: Slot [7] registered [ 0.239121] acpiphp: Slot [8] registered [ 0.240142] acpiphp: Slot [9] registered [ 0.242160] acpiphp: Slot [10] registered [ 0.243137] acpiphp: Slot [3] registered [ 0.245122] acpiphp: Slot [4] registered [ 0.246095] acpiphp: Slot [11] registered [ 0.247094] acpiphp: Slot [12] registered [ 0.248079] acpiphp: Slot [13] registered [ 0.249043] acpiphp: Slot [14] registered [ 0.250105] acpiphp: Slot [15] registered [ 0.252122] acpiphp: Slot [16] registered [ 0.253080] acpiphp: Slot [17] registered [ 0.254090] acpiphp: Slot [18] registered [ 0.255098] acpiphp: Slot [19] registered [ 0.256115] acpiphp: Slot [20] registered [ 0.258101] acpiphp: Slot [21] registered [ 0.259126] acpiphp: Slot [22] registered [ 0.261103] acpiphp: Slot [23] registered [ 0.262298] acpiphp: Slot [24] registered [ 0.264124] acpiphp: Slot [25] registered [ 0.266166] acpiphp: Slot [26] registered [ 0.267081] acpiphp: Slot [27] registered [ 0.268122] acpiphp: Slot [28] registered [ 0.270184] acpiphp: Slot [29] registered [ 0.271111] acpiphp: Slot [30] registered [ 0.273150] acpiphp: Slot [31] registered [ 0.274076] PCI host bridge to bus 0000:00 [ 0.276021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.278036] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.281027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.283027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.287028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.290059] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.293568] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.297217] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.301442] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.310695] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.318747] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.321023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.324023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.327028] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.331121] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.333842] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.336047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.339935] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.346016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.359018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.365017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.370931] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.382020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.390022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.407021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.419460] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.429016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.440022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.454022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.462344] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.471017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.477019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.499118] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.522332] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.530014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.535019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.549022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.556689] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.567015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.577021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.593021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.602194] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.607023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.612016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.624019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.634125] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.637361] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.639236] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.641323] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.644236] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.650053] iommu: Default domain type: Passthrough [ 0.651346] SCSI subsystem initialized [ 0.652104] ACPI: bus type USB registered [ 0.652946] usbcore: registered new interface driver usbfs [ 0.654057] usbcore: registered new interface driver hub [ 0.655064] usbcore: registered new device driver usb [ 0.656120] pps_core: LinuxPPS API ver. 1 registered [ 0.657006] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.659055] PTP clock support registered [ 0.661088] EDAC MC: Ver: 3.0.0 [ 0.663115] PCI: Using ACPI for IRQ routing [ 0.665877] NetLabel: Initializing [ 0.667013] NetLabel: domain hash size = 128 [ 0.668010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.670095] NetLabel: unlabeled traffic allowed by default [ 0.672168] vgaarb: loaded [ 0.674214] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.675013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.685250] clocksource: Switched to clocksource kvm-clock [ 0.791829] VFS: Disk quotas dquot_6.6.0 [ 0.793160] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.795409] *** VALIDATE ramfs *** [ 0.796636] *** VALIDATE hugetlbfs *** [ 0.798244] pnp: PnP ACPI init [ 0.800663] pnp: PnP ACPI: found 6 devices [ 0.828375] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.832079] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.834541] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.837196] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.840017] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.842671] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.845736] NET: Registered protocol family 2 [ 0.848409] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.853543] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.857056] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.862377] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.865801] TCP: Hash tables configured (established 65536 bind 65536) [ 0.868682] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.871543] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.874787] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.877536] NET: Registered protocol family 1 [ 0.879821] RPC: Registered named UNIX socket transport module. [ 0.881701] RPC: Registered udp transport module. [ 0.883314] RPC: Registered tcp transport module. [ 0.885233] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.887445] NET: Registered protocol family 44 [ 0.889066] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.891044] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.892656] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.895034] PCI: CLS 0 bytes, default 64 [ 0.896630] Unpacking initramfs... [ 2.302678] debug: unmapping init [mem 0xffff8f1a3cc54000-0xffff8f1a3ffbffff] [ 2.306424] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.308529] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.311366] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.799927] Initialise system trusted keyrings [ 2.801558] Key type blacklist registered [ 2.803373] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.813080] zbud: loaded [ 2.816916] *** VALIDATE nfs *** [ 2.818064] *** VALIDATE nfs4 *** [ 2.819191] pstore: using deflate compression [ 2.822363] Platform Keyring initialized [ 2.928663] NET: Registered protocol family 38 [ 2.930546] Key type asymmetric registered [ 2.931981] Asymmetric key parser 'x509' registered [ 2.933852] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.936654] io scheduler mq-deadline registered [ 2.938277] io scheduler kyber registered [ 2.939822] io scheduler bfq registered [ 2.941628] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.944359] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.946793] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.949427] ACPI: Power Button [PWRF] [ 2.954375] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.960494] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.975538] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.988281] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.004241] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.033591] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.062654] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.068767] Non-volatile memory driver v1.3 [ 3.070283] Linux agpgart interface v0.103 [ 3.106149] virtio_blk virtio1: [vda] 139480 512-byte logical blocks (71.4 MB/68.1 MiB) [ 3.108532] vda: detected capacity change from 0 to 71413760 [ 3.123623] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.126508] vdb: detected capacity change from 0 to 1073741824 [ 3.145635] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.148122] vdc: detected capacity change from 0 to 2621440000 [ 3.161586] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.163608] vdd: detected capacity change from 0 to 2621440000 [ 3.178109] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.180076] vde: detected capacity change from 0 to 4294967296 [ 3.192116] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.194405] vdf: detected capacity change from 0 to 4294967296 [ 3.199230] libphy: Fixed MDIO Bus: probed [ 3.216787] usbcore: registered new interface driver usbserial_generic [ 3.218863] usbserial: USB Serial support registered for generic [ 3.220850] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.223985] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.225135] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.226610] mousedev: PS/2 mouse device common for all mice [ 3.228782] rtc_cmos 00:05: RTC can wake from S4 [ 3.230318] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.232808] rtc_cmos 00:05: registered as rtc0 [ 3.232848] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.232875] intel_pstate: CPU model not supported [ 3.236398] hid: raw HID events driver (C) Jiri Kosina [ 3.239793] usbcore: registered new interface driver usbhid [ 3.241096] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.242189] usbhid: USB HID core driver [ 3.242335] drop_monitor: Initializing network drop monitor service [ 3.246163] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.247451] Initializing XFRM netlink socket [ 3.253154] NET: Registered protocol family 10 [ 3.255849] Segment Routing with IPv6 [ 3.257088] NET: Registered protocol family 17 [ 3.258734] mpls_gso: MPLS GSO support [ 3.264264] RAS: Correctable Errors collector initialized. [ 3.266319] AVX version of gcm_enc/dec engaged. [ 3.267480] AES CTR mode by8 optimization enabled [ 3.326776] sched_clock: Marking stable (3326743604, 0)->(4190005998, -863262394) [ 3.329741] registered taskstats version 1 [ 3.330881] Loading compiled-in X.509 certificates [ 3.331841] zswap: loaded using pool lzo/zbud [ 3.349844] Key type big_key registered [ 3.362049] Key type encrypted registered [ 3.363600] ima: No TPM chip found, activating TPM-bypass! [ 3.365463] ima: Allocated hash algorithm: sha1 [ 3.366970] ima: No architecture policies found [ 3.367910] evm: Initialising EVM extended attributes: [ 3.369071] evm: security.selinux [ 3.369612] evm: security.ima [ 3.370272] evm: security.capability [ 3.370970] evm: HMAC attrs: 0x1 [ 3.372675] rtc_cmos 00:05: setting system clock to 2026-06-12 13:48:51 UTC (1781272131) [ 3.376603] debug: unmapping init [mem 0xffffffffab003000-0xffffffffab1fffff] [ 3.378949] debug: unmapping init [mem 0xffffffffa9d82000-0xffffffffaa058fff] [ 3.387069] Write protecting the kernel read-only data: 28672k [ 3.389199] debug: unmapping init [mem 0xffffffffa8403000-0xffffffffa85fffff] [ 3.391147] debug: unmapping init [mem 0xffffffffa8d14000-0xffffffffa8dfffff] [ 3.417843] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.423420] systemd[1]: Detected virtualization kvm. [ 3.424807] systemd[1]: Detected architecture x86-64. [ 3.425993] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.448025] systemd[1]: No hostname configured. [ 3.449499] systemd[1]: Set hostname to . [ 3.451358] random: systemd: uninitialized urandom read (16 bytes read) [ 3.453473] systemd[1]: Initializing machine ID from random generator. [ 3.491833] random: ln: uninitialized urandom read (6 bytes read) [ 3.580820] random: systemd: uninitialized urandom read (16 bytes read) [ 3.583326] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.589483] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ 3.592819] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Reached target Initrd Root Device. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.232609] device-mapper: uevent: version 1.0.3 [ 4.234158] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ 4.452444] random: fast init done [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.858434] virtio_net virtio0 ens2: renamed from eth0 [ 4.937799] scsi host0: ata_piix [ 4.949433] scsi host1: ata_piix [ 4.951017] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.953482] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.549450] dracut-initqueue[582]: RTNETLINK answers: File exists [ 9.875461] random: crng init done [ 9.876532] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.141584] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.271179] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.544448] SELinux: Disabled at runtime. [ 11.606697] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.614765] systemd[1]: Detected virtualization kvm. [ 11.616402] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.007092] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.009569] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.013442] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.017370] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.020173] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.026205] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.030392] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ 12.060568] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ 12.375461] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /mnt. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.699448] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.739548] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.844033] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.861123] EDAC sbridge: Ver: 1.1.2 [ 13.997085] Key type dns_resolver registered [ 14.302498] NFS: Registering the id_resolver key type [ 14.304477] Key type id_resolver registered [ 14.305856] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg129-server login: [ 60.645906] libcfs: loading out-of-tree module taints kernel. [ 60.712757] Key type ._llcrypt registered [ 60.716373] Key type .llcrypt registered [ 60.879405] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_hostid [ 83.217032] hrtimer: interrupt took 6876544 ns [ 84.571799] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 86.198529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 86.213657] alg: No test for adler32 (adler32-zlib) [ 87.567724] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 88.261961] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 89.959254] Key type lgssc registered [ 92.248111] Lustre: Echo OBD driver; http://www.lustre.org/ [ 113.700865] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 162.759326] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 177.966509] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 177.999884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 179.389871] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 179.445543] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 179.624315] Lustre: lustre-MDT0000: new disk, initializing [ 179.805542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 179.841920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 187.475636] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 204.964740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 205.184987] Lustre: 6508:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 205.300684] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 205.315994] Lustre: Skipped 1 previous similar message [ 205.597054] Lustre: lustre-MDT0001: new disk, initializing [ 205.786966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 205.848570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 205.865221] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 213.056386] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 218.747681] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 230.757943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 231.070691] Lustre: lustre-OST0000: new disk, initializing [ 231.074459] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 231.079105] Lustre: 8449:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 231.134070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 231.371110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 231.389352] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 231.527202] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 240.237187] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 260.189563] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 260.535814] Lustre: lustre-OST0001: new disk, initializing [ 260.555689] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 260.562300] Lustre: 9523:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 260.709735] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 267.982030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 267.993699] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 268.142483] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 268.157103] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 282.020764] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 287.093814] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 294.820654] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing check_logdir /tmp/testlogs/ [ 299.977951] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing yml_node [ 305.587610] Lustre: DEBUG MARKER: Client: 2.17.54.4 [ 308.817446] Lustre: DEBUG MARKER: MDS: 2.17.54.4 [ 312.723622] Lustre: DEBUG MARKER: OSS: 2.17.54.4 [ 315.144090] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jun 12 09:54:01 EDT 2026 [ 334.479884] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 336.463528] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 354.272769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 354.278653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 354.300694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 355.299626] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 355.301350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 355.326569] Lustre: Skipped 2 previous similar messages [ 359.625190] Lustre: server umount lustre-MDT0000 complete [ 364.255744] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781272492 with bad export cookie 9639100742477960030 [ 364.258864] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 364.267072] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 365.536081] LustreError: 6517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 365.540588] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 365.544477] LustreError: 6517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 365.555062] Lustre: Skipped 2 previous similar messages [ 365.567036] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 365.574214] Lustre: Skipped 1 previous similar message [ 369.568349] LustreError: 6516:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 369.571652] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 369.600878] Lustre: Skipped 1 previous similar message [ 370.725131] Lustre: server umount lustre-MDT0001 complete [ 375.203608] Lustre: server umount lustre-OST0000 complete [ 379.520352] Lustre: server umount lustre-OST0001 complete [ 387.234935] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_hostid [ 399.758468] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 443.661053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 444.009155] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 444.054914] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 444.156157] Lustre: lustre-MDT0000: new disk, initializing [ 444.244821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 444.263690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 449.871404] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 462.499771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 462.666077] Lustre: 15081:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 462.760392] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 462.783445] Lustre: Skipped 1 previous similar message [ 462.884308] Lustre: lustre-MDT0001: new disk, initializing [ 463.047638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 463.094727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 463.115357] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 468.201596] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 478.780635] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 486.048579] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 495.173480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 495.530554] Lustre: lustre-OST0000: new disk, initializing [ 495.535939] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 495.552755] Lustre: 17051:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 495.712223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 497.231699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 497.245616] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 497.396901] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 502.996376] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 513.900462] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 519.615580] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 519.947323] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 525.013450] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 525.177411] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 528.357770] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 528.362977] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 528.389847] Lustre: Skipped 1 previous similar message [ 528.390627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 533.854807] Lustre: server umount lustre-OST0000 complete [ 540.129874] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 540.142544] LustreError: Skipped 1 previous similar message [ 540.157609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 540.178218] Lustre: Skipped 1 previous similar message [ 540.188724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 540.201792] Lustre: Skipped 3 previous similar messages [ 545.350182] Lustre: server umount lustre-MDT0000 complete [ 548.837575] LustreError: 15094:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 548.865283] LustreError: 15094:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 550.540619] LustreError: 15073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781272678 with bad export cookie 9639100742477961570 [ 550.541564] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 550.568350] LustreError: 15073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 550.861625] Lustre: server umount lustre-MDT0001 complete [ 556.420828] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 09:58:02 (1781272682) [ 564.501161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 564.953511] LustreError: 19317:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 565.040078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 570.336625] LustreError: 19318:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 570.355849] LustreError: 19318:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 570.632664] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 580.359429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 580.581199] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 580.592791] Lustre: Skipped 2 previous similar messages [ 586.272490] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 595.065855] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 601.231505] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 610.645731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 620.320843] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 631.029118] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 645.091725] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 645.095865] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 645.100233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 645.116203] Lustre: Skipped 2 previous similar messages [ 651.402129] Lustre: server umount lustre-OST0000 complete [ 657.889706] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 662.302990] Lustre: server umount lustre-MDT0000 complete [ 665.569500] LustreError: 19317:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 665.601333] LustreError: 19317:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 666.871837] LustreError: 19299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781272794 with bad export cookie 9639100742477962452 [ 666.887493] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 666.893182] LustreError: 19299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 667.348729] Lustre: server umount lustre-MDT0001 complete [ 682.812836] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 687.480714] Key type lgssc unregistered [ 688.002561] LNet: 22872:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 688.016467] LNetError: 22872:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 689.061491] LNet: Removed LNI 192.168.201.129@tcp [ 690.323903] Key type .llcrypt unregistered [ 690.325545] Key type ._llcrypt unregistered [ 716.204538] Key type ._llcrypt registered [ 716.206571] Key type .llcrypt registered [ 716.284793] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 717.443245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 717.485943] alg: No test for adler32 (adler32-zlib) [ 718.754743] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 719.133616] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 720.903811] Key type lgssc registered [ 722.196773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 730.367804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 730.422214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 732.087614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 737.442764] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 742.372452] LustreError: 24290:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 742.383022] LustreError: 24290:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 746.431901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 747.033170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 753.094757] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 761.874682] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 768.186785] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 776.969214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 777.690906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 782.849748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 784.875361] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 792.861563] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 811.492258] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 811.492828] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 811.500533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 811.517461] Lustre: Skipped 1 previous similar message [ 816.620272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 816.629092] Lustre: Skipped 1 previous similar message [ 817.741976] Lustre: server umount lustre-OST0000 complete [ 823.777995] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 823.783325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 823.797567] Lustre: Skipped 1 previous similar message [ 823.810956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 826.853606] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 826.857900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 826.878323] Lustre: Skipped 1 previous similar message [ 828.952590] Lustre: server umount lustre-MDT0000 complete [ 831.971054] LustreError: 24289:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 831.993847] LustreError: 24289:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 833.029045] LustreError: 24271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781272961 with bad export cookie 15312347365041459322 [ 833.033451] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 833.045793] LustreError: 24271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 833.607690] Lustre: server umount lustre-MDT0001 complete [ 846.049125] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 848.822312] Key type lgssc unregistered [ 849.060621] LNet: 28280:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 849.072028] LNetError: 28280:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 849.115567] LNet: Removed LNI 192.168.201.129@tcp [ 850.185248] Key type .llcrypt unregistered [ 850.187665] Key type ._llcrypt unregistered [ 875.428738] Key type ._llcrypt registered [ 875.435600] Key type .llcrypt registered [ 875.624118] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 877.168674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 877.203465] alg: No test for adler32 (adler32-zlib) [ 878.318265] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 878.567887] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 880.247327] Key type lgssc registered [ 881.541883] Lustre: Echo OBD driver; http://www.lustre.org/ [ 890.281111] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 890.337241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 891.965706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 896.760886] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 902.118073] LustreError: 29696:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 902.129920] LustreError: 29696:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 905.600044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 906.001561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 911.174382] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 919.409654] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 925.517453] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 940.450310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 940.917877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 948.673322] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 956.804779] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 963.474665] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 976.866808] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 976.881684] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 976.890811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 978.917102] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 978.924035] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 978.927242] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 978.952574] Lustre: Skipped 1 previous similar message [ 981.264639] Lustre: server umount lustre-OST0000 complete [ 988.130607] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 988.134072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 988.182563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 992.799296] Lustre: server umount lustre-MDT0000 complete [ 994.280266] LustreError: 29692:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 994.303713] LustreError: 29692:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 997.086639] LustreError: 29675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781273125 with bad export cookie 17029310159530275436 [ 997.091222] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 997.100026] LustreError: 29675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 997.461438] Lustre: server umount lustre-MDT0001 complete [ 1011.546786] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 1014.263775] Key type lgssc unregistered [ 1014.616767] LNet: 33599:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1014.623826] LNetError: 33599:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1015.656517] LNet: Removed LNI 192.168.201.129@tcp [ 1016.606897] Key type .llcrypt unregistered [ 1016.610433] Key type ._llcrypt unregistered [ 1045.390996] Key type ._llcrypt registered [ 1045.403676] Key type .llcrypt registered [ 1045.633516] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 1047.368257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1047.457410] alg: No test for adler32 (adler32-zlib) [ 1048.868984] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 1049.390465] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 1051.223199] Key type lgssc registered [ 1052.364272] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1060.700988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1060.732030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1062.205934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1067.821755] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 1072.610650] LustreError: 35011:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1072.635563] LustreError: 35011:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1077.739365] LustreError: 35010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1078.068550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1078.587927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1084.174232] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 1094.946226] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1101.521623] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1110.496967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1111.053206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1118.311350] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 1127.100731] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1132.591516] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1142.763608] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1142.772236] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1142.778206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1142.788632] Lustre: Skipped 1 previous similar message [ 1144.467279] Lustre: server umount lustre-OST0000 complete [ 1150.432600] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1150.439389] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1150.452406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1150.454186] Lustre: Skipped 1 previous similar message [ 1152.999892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1153.000767] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1154.423520] Lustre: server umount lustre-MDT0000 complete [ 1158.116485] LustreError: 36538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1158.143365] LustreError: 36538:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1158.937947] LustreError: 34991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781273287 with bad export cookie 11603845692849586718 [ 1158.943150] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1158.950514] LustreError: 34991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1159.337961] Lustre: server umount lustre-MDT0001 complete [ 1171.543227] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 1174.537885] Key type lgssc unregistered [ 1174.842678] LNet: 38574:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1174.855747] LNetError: 38574:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1175.918914] LNet: Removed LNI 192.168.201.129@tcp [ 1176.981845] Key type .llcrypt unregistered [ 1176.984437] Key type ._llcrypt unregistered [ 1203.693148] Key type ._llcrypt registered [ 1203.694801] Key type .llcrypt registered [ 1203.907553] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 1205.126906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1205.292611] alg: No test for adler32 (adler32-zlib) [ 1206.370751] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 1206.630188] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 1208.407210] Key type lgssc registered [ 1209.720684] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1219.383814] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1219.398302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1221.032705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1226.070177] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 1231.329812] LustreError: 39991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1231.366808] LustreError: 39991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1236.455747] LustreError: 39990:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1237.186736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1237.645797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1243.500069] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 1252.385368] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1260.363320] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1268.922130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1269.322235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1277.415934] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 1287.297197] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1291.894166] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1299.205461] Lustre: lustre-OST0000: Not available for connect from 192.168.201.29@tcp (stopping) [ 1302.504838] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1302.505920] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1302.522114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1302.550410] Lustre: Skipped 1 previous similar message [ 1304.305562] Lustre: lustre-OST0000: Not available for connect from 192.168.201.29@tcp (stopping) [ 1304.321927] Lustre: Skipped 1 previous similar message [ 1304.932016] Lustre: server umount lustre-OST0000 complete [ 1326.588402] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1326.606161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1326.617130] Lustre: Skipped 1 previous similar message [ 1331.683198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1331.695365] Lustre: Skipped 2 previous similar messages [ 1333.335920] Lustre: server umount lustre-MDT0000 complete [ 1336.814701] LustreError: 39991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1336.868752] LustreError: 39991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1337.733159] LustreError: 39969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781273465 with bad export cookie 8120704289801262289 [ 1337.750770] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1337.761093] LustreError: 39969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1352.159891] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1352.479403] Lustre: server umount lustre-MDT0001 complete [ 1364.307599] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 1367.693866] Key type lgssc unregistered [ 1368.072138] LNet: 43700:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1368.078858] LNetError: 43700:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1369.125701] LNet: Removed LNI 192.168.201.129@tcp [ 1370.221217] Key type .llcrypt unregistered [ 1370.225079] Key type ._llcrypt unregistered [ 1396.046180] Key type ._llcrypt registered [ 1396.048688] Key type .llcrypt registered [ 1396.251031] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 1397.734707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1397.952699] alg: No test for adler32 (adler32-zlib) [ 1399.648546] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 1400.104973] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 1401.975211] Key type lgssc registered [ 1403.675245] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1414.085915] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1414.125638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1415.738960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1421.171659] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 1425.902274] LustreError: 45115:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1425.912436] LustreError: 45115:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1431.021359] LustreError: 45115:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1431.118619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1431.728292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1436.814192] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 1445.719640] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1452.066533] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1460.346746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1460.677916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1461.778640] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1467.796778] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 1476.629208] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1484.569525] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.29@tcp (stopping) [ 1486.306776] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1486.313044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1486.332226] Lustre: Skipped 1 previous similar message [ 1489.655678] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.29@tcp (stopping) [ 1489.663311] Lustre: Skipped 2 previous similar messages [ 1490.595079] Lustre: server umount lustre-MDT0000 complete [ 1491.436467] LustreError: 45115:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1491.464210] LustreError: 45115:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1494.679453] LustreError: 45094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781273622 with bad export cookie 15809640689847533297 [ 1494.682638] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1494.693883] LustreError: 45094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1494.765514] LustreError: 45111:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1494.798640] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.29@tcp (stopping) [ 1496.547030] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1496.584294] Lustre: Skipped 1 previous similar message [ 1499.554295] LustreError: 45111:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1499.554332] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1499.566344] LustreError: 45111:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1499.579631] Lustre: Skipped 1 previous similar message [ 1501.184951] Lustre: server umount lustre-MDT0001 complete [ 1530.847092] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1531.147229] Lustre: server umount lustre-OST0000 complete [ 1552.038266] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 1555.401690] Key type lgssc unregistered [ 1555.682625] LNet: 48977:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1555.695075] LNetError: 48977:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1555.728629] LNet: Removed LNI 192.168.201.129@tcp [ 1556.798814] Key type .llcrypt unregistered [ 1556.800567] Key type ._llcrypt unregistered [ 1588.170490] Key type ._llcrypt registered [ 1588.173303] Key type .llcrypt registered [ 1588.269908] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 1590.241600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1590.302721] alg: No test for adler32 (adler32-zlib) [ 1591.538366] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 1591.864173] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 1593.599373] Key type lgssc registered [ 1595.350167] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1605.043294] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1605.055270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1620.578809] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1625.695255] LustreError: 50365:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.129@tcp: failed processing log, type 4: rc = -110 [ 1652.447432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1660.321084] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 1668.863242] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1834.962212] Lustre: server umount lustre-OST0000 complete [ 1852.693586] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 1855.162771] Key type lgssc unregistered [ 1855.377277] LNet: 52042:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1855.384330] LNetError: 52042:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1855.394844] LNet: Removed LNI 192.168.201.129@tcp [ 1856.248582] Key type .llcrypt unregistered [ 1856.255205] Key type ._llcrypt unregistered [ 1884.923568] Key type ._llcrypt registered [ 1884.925937] Key type .llcrypt registered [ 1885.104714] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 1886.310143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1886.469792] alg: No test for adler32 (adler32-zlib) [ 1887.696519] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 1887.994841] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 1889.833340] Key type lgssc registered [ 1891.474915] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1900.646680] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1900.691308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1902.317860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1907.886858] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 1912.802477] LustreError: 53459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1912.823787] LustreError: 53459:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1917.933205] LustreError: 53460:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1918.695550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1919.437760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1924.697427] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 1932.478349] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1939.116859] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1946.838727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1947.110287] LustreError: 55280:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1947.318238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1952.761033] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1954.780581] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 1963.133424] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1968.096123] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1968.110669] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1968.136572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1969.121847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1969.121868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1969.160227] Lustre: Skipped 1 previous similar message [ 1971.915680] Lustre: server umount lustre-OST0000 complete [ 1979.363087] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1979.370384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1979.371295] Lustre: Skipped 1 previous similar message [ 1979.390184] Lustre: Skipped 1 previous similar message [ 1982.244714] Lustre: server umount lustre-MDT0000 complete [ 1984.492497] LustreError: 53455:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1984.528487] LustreError: 53455:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1986.544291] LustreError: 53439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781274114 with bad export cookie 3631330664096758011 [ 1986.551161] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1986.552271] LustreError: 53439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1987.282454] Lustre: server umount lustre-MDT0001 complete [ 2000.310425] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 2003.138644] Key type lgssc unregistered [ 2003.456675] LNet: 56996:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2003.464890] LNetError: 56996:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2003.481289] LNet: Removed LNI 192.168.201.129@tcp [ 2004.235180] Key type .llcrypt unregistered [ 2004.236823] Key type ._llcrypt unregistered [ 2029.441913] Key type ._llcrypt registered [ 2029.446310] Key type .llcrypt registered [ 2029.587041] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 2030.622191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2030.747081] alg: No test for adler32 (adler32-zlib) [ 2031.947805] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 2032.312990] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 2034.055438] Key type lgssc registered [ 2035.054230] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2042.656258] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2042.694208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2058.271553] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2063.391414] LustreError: 58382:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.129@tcp: failed processing log, type 4: rc = -110 [ 2090.079497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2096.428283] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2104.713499] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2112.801129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2113.603692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2113.707902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2118.625386] LustreError: 58408:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2118.643888] LustreError: 58408:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2119.654587] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2123.749124] LustreError: 58407:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2124.797456] LustreError: 59417:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2124.821687] LustreError: 59417:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2129.892674] LustreError: 58408:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2129.916553] LustreError: 58408:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2129.999767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2130.339343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2135.948626] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2143.556652] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2148.932855] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2154.477782] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2154.485432] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2154.517763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2156.000975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2156.018314] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2156.034307] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2157.592773] Lustre: server umount lustre-OST0000 complete [ 2181.602896] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2181.606571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2181.614385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2181.616686] Lustre: Skipped 1 previous similar message [ 2184.672641] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2184.682311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2184.688109] Lustre: Skipped 1 previous similar message [ 2187.705055] Lustre: server umount lustre-MDT0000 complete [ 2189.803402] LustreError: 59420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2189.823957] LustreError: 59420:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2192.318490] LustreError: 58388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781274320 with bad export cookie 2885082501590979143 [ 2192.325562] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2192.345428] LustreError: 58388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2198.847197] Lustre: server umount lustre-MDT0001 complete [ 2211.463724] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 2215.134869] Key type lgssc unregistered [ 2215.395087] LNet: 62129:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2215.406375] LNetError: 62129:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2215.423358] LNet: Removed LNI 192.168.201.129@tcp [ 2216.475804] Key type .llcrypt unregistered [ 2216.482576] Key type ._llcrypt unregistered [ 2242.612667] Key type ._llcrypt registered [ 2242.616923] Key type .llcrypt registered [ 2242.734457] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 2243.984647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2244.074948] alg: No test for adler32 (adler32-zlib) [ 2245.276515] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 2245.586472] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 2247.287221] Key type lgssc registered [ 2248.482658] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2257.477210] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2257.520242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2259.090434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2263.838274] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2269.671529] LustreError: 63545:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2269.688257] LustreError: 63545:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2273.549504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2274.157066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2279.726814] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2288.887473] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2296.033489] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2303.826984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2304.003625] LustreError: 65370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2304.318839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2309.647676] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2311.072462] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2318.609903] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2336.235605] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2336.237253] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2336.250898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2336.261846] Lustre: Skipped 1 previous similar message [ 2337.690951] Lustre: server umount lustre-OST0000 complete [ 2345.952914] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2345.966891] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2345.988276] Lustre: Skipped 1 previous similar message [ 2346.001498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2351.584986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2351.597611] Lustre: Skipped 3 previous similar messages [ 2356.192484] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2356.363648] Lustre: server umount lustre-MDT0000 complete [ 2356.710109] LustreError: 63545:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2360.548144] LustreError: 63525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781274488 with bad export cookie 1645436429574437191 [ 2360.568130] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2360.571217] LustreError: 63525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2361.022667] Lustre: server umount lustre-MDT0001 complete [ 2371.675423] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 2374.386379] Key type lgssc unregistered [ 2374.687683] LNet: 67103:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2374.701344] LNetError: 67103:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2374.733482] LNet: Removed LNI 192.168.201.129@tcp [ 2375.636976] Key type .llcrypt unregistered [ 2375.639303] Key type ._llcrypt unregistered [ 2400.616517] Key type ._llcrypt registered [ 2400.618560] Key type .llcrypt registered [ 2400.733722] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 2401.592953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2401.674399] alg: No test for adler32 (adler32-zlib) [ 2402.882388] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 2403.114283] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 2404.775761] Key type lgssc registered [ 2406.108767] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2415.217609] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 10:29:01 (1781274541) [ 2422.300345] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2422.322042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2423.774419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2428.439444] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2432.994960] LustreError: 68785:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2433.006860] LustreError: 68785:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2438.023615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2438.119087] LustreError: 68784:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2438.380487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2443.858983] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2453.067670] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2460.005396] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2468.197980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2468.333565] LustreError: 70607:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2468.724737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2473.986553] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2476.274949] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2485.795118] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2493.011216] Lustre: Failing over lustre-MDT0000 [ 2493.381859] Lustre: server umount lustre-MDT0000 complete [ 2493.410431] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2493.435053] LustreError: 68779:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2493.464426] LustreError: 68779:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2498.534347] LustreError: 68781:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2498.565894] LustreError: 68781:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2505.496903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2505.614127] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2505.695754] Lustre: *** cfs_fail_loc=135, val=0*** [ 2505.759614] LustreError: 71875:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2505.762947] LustreError: 71875:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.129@tcp: cfg command failed: rc = -2 [ 2505.768983] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2505.768983] [ 2505.773331] LustreError: MGC192.168.201.129@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2505.780956] LustreError: 71863:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2505.785676] LustreError: 71863:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 2505.790620] LustreError: 71863:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 2505.795974] LustreError: 71863:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2505.812117] Lustre: server umount lustre-MDT0000 complete [ 2505.813419] LustreError: 71863:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2508.772910] LustreError: 68785:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2508.804421] LustreError: 68785:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2513.043959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2513.146084] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2513.463923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2513.515601] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2514.161300] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2518.230020] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2518.506245] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2518.548647] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2518.614150] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2528.747043] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2528.758755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2528.759530] Lustre: Skipped 3 previous similar messages [ 2528.774729] Lustre: Skipped 1 previous similar message [ 2533.286582] Lustre: server umount lustre-OST0000 complete [ 2538.976794] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2538.984872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2538.999832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2539.023668] Lustre: Skipped 1 previous similar message [ 2543.680553] Lustre: server umount lustre-MDT0000 complete [ 2544.096454] LustreError: 68785:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2544.113395] LustreError: 68785:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2548.100593] LustreError: 68767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781274676 with bad export cookie 6811754669411829202 [ 2548.130639] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2548.369372] Lustre: server umount lustre-MDT0001 complete [ 2559.414724] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 2562.490395] Key type lgssc unregistered [ 2562.809316] LNet: 73657:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2562.812996] LNetError: 73657:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2562.834136] LNet: Removed LNI 192.168.201.129@tcp [ 2563.661927] Key type .llcrypt unregistered [ 2563.664121] Key type ._llcrypt unregistered [ 2586.734698] Key type ._llcrypt registered [ 2586.739173] Key type .llcrypt registered [ 2586.900756] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 2588.332067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2588.365225] alg: No test for adler32 (adler32-zlib) [ 2589.702282] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 2589.959861] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 2591.599692] Key type lgssc registered [ 2592.977075] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2600.538347] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2600.551568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2601.929493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2605.961118] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2612.193565] LustreError: 75049:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2612.202567] LustreError: 75049:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2614.145863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2614.530915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2619.177709] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2627.282465] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2633.181477] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2640.048853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2640.480895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2646.421988] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2647.663715] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2654.714662] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2664.966069] Lustre: Failing over lustre-MDT0000 [ 2665.279132] Lustre: server umount lustre-MDT0000 complete [ 2665.953749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2665.962335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2665.975985] LustreError: 75049:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2667.805529] LustreError: 75044:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2669.551925] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2669.563816] Lustre: Skipped 1 previous similar message [ 2672.881630] LustreError: 76249:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2672.911186] LustreError: 76249:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2675.526306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2675.664628] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2675.807710] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2675.957104] LustreError: 78137:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2675.963301] LustreError: 78137:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.129@tcp: cfg command failed: rc = -9 [ 2675.970142] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2675.970142] [ 2675.977100] LustreError: MGC192.168.201.129@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2675.986722] LustreError: 78125:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2676.004317] LustreError: 78125:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -9 [ 2676.031752] LustreError: 78125:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 2676.041061] LustreError: 78125:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2676.074965] Lustre: server umount lustre-MDT0000 complete [ 2676.078091] LustreError: 78125:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2678.002968] LustreError: 75046:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2678.023994] LustreError: 75046:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2683.159834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2683.303811] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2683.319519] LustreError: 78484:0:(ldlm_resource.c:1179:ldlm_resource_complain()) MGC192.168.201.129@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000f216e7f9) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2683.342729] LustreError: 75042:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2683.748881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2683.834497] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2684.218732] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2688.322370] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2689.015654] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2689.041697] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2689.104713] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2699.243284] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2699.247601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2699.264525] Lustre: Skipped 1 previous similar message [ 2699.275427] Lustre: Skipped 1 previous similar message [ 2702.323372] Lustre: server umount lustre-OST0000 complete [ 2709.473555] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2709.487631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2709.492092] Lustre: Skipped 1 previous similar message [ 2709.494235] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2709.511731] Lustre: Skipped 1 previous similar message [ 2714.592645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2714.606749] Lustre: Skipped 1 previous similar message [ 2719.712920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2719.717682] Lustre: Skipped 1 previous similar message [ 2720.223197] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2720.538808] Lustre: server umount lustre-MDT0000 complete [ 2724.721196] LustreError: 75032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781274852 with bad export cookie 527137483677735949 [ 2724.727278] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2724.735763] LustreError: 75032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2725.203942] Lustre: server umount lustre-MDT0001 complete [ 2738.512480] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 2742.374899] Key type lgssc unregistered [ 2742.680400] LNet: 79918:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2742.683822] LNetError: 79918:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2743.719360] LNet: Removed LNI 192.168.201.129@tcp [ 2744.691888] Key type .llcrypt unregistered [ 2744.693937] Key type ._llcrypt unregistered [ 2760.564571] Key type ._llcrypt registered [ 2760.568478] Key type .llcrypt registered [ 2760.682781] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 2760.985622] Key type .llcrypt unregistered [ 2760.999384] Key type ._llcrypt unregistered [ 2785.079439] Key type ._llcrypt registered [ 2785.084800] Key type .llcrypt registered [ 2785.211487] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 2787.038512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2787.098319] alg: No test for adler32 (adler32-zlib) [ 2788.376329] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 2788.762250] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 2790.519175] Key type lgssc registered [ 2792.621970] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2804.810767] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2804.842477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2806.342770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2811.722561] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2816.485607] LustreError: 81821:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2816.498914] LustreError: 81821:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2821.606207] LustreError: 81822:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2822.034667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2822.520779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2827.133271] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2835.800425] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2841.724062] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2849.249614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2849.692641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2851.772530] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2857.079130] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2865.313595] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2878.441817] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2878.443253] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2878.447042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2878.533411] Lustre: Skipped 1 previous similar message [ 2883.715161] LustreError: 83644:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2883.739687] LustreError: 83644:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2883.839782] Lustre: server umount lustre-OST0000 complete [ 2889.184497] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2889.188576] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2889.197411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2889.227324] Lustre: Skipped 1 previous similar message [ 2889.236073] Lustre: Skipped 2 previous similar messages [ 2894.314538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2894.486218] Lustre: server umount lustre-MDT0000 complete [ 2899.155934] LustreError: 81804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781275027 with bad export cookie 6159243639346982110 [ 2899.170474] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2899.174932] LustreError: 81804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2899.557095] Lustre: server umount lustre-MDT0001 complete [ 2910.803953] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 2913.761029] Key type lgssc unregistered [ 2914.124404] LNet: 85356:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2914.136559] LNetError: 85356:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2914.166915] LNet: Removed LNI 192.168.201.129@tcp [ 2915.217860] Key type .llcrypt unregistered [ 2915.224078] Key type ._llcrypt unregistered [ 2941.338716] Key type ._llcrypt registered [ 2941.342630] Key type .llcrypt registered [ 2941.503336] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 2942.740824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2942.900469] alg: No test for adler32 (adler32-zlib) [ 2944.031682] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 2944.216663] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 2945.871915] Key type lgssc registered [ 2947.350786] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2954.920622] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2954.929731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2956.646702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2961.114694] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2965.994918] LustreError: 86785:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2966.016725] LustreError: 86785:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2969.878653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2970.228787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2975.001748] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 2983.060500] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2988.891557] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2995.418041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2995.713694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3001.913460] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 3002.542154] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3010.748178] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3020.634932] Lustre: server umount lustre-OST0000 complete [ 3025.380473] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3025.385455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3025.392550] Lustre: Skipped 1 previous similar message [ 3030.498422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3030.513039] Lustre: Skipped 2 previous similar messages [ 3034.677706] Lustre: server umount lustre-MDT0000 complete [ 3035.620589] LustreError: 86786:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3035.653981] LustreError: 86786:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3039.511557] LustreError: 86765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781275167 with bad export cookie 14334864193711751010 [ 3039.520535] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3039.524486] LustreError: 86765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3049.172791] Lustre: server umount lustre-MDT0001 complete [ 3061.752492] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 3065.280704] Key type lgssc unregistered [ 3065.578977] LNet: 90317:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3065.583906] LNetError: 90317:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3065.597650] LNet: Removed LNI 192.168.201.129@tcp [ 3066.785224] Key type .llcrypt unregistered [ 3066.793447] Key type ._llcrypt unregistered [ 3097.572883] Key type ._llcrypt registered [ 3097.585986] Key type .llcrypt registered [ 3097.805404] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 3099.141247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3099.164770] alg: No test for adler32 (adler32-zlib) [ 3100.271067] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 3100.558806] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 3102.289361] Key type lgssc registered [ 3103.363222] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3113.764329] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3113.798709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3115.264605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3121.219770] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3125.730551] LustreError: 91733:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3125.748418] LustreError: 91733:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3130.849049] LustreError: 91734:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3132.124190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3132.727578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3139.054946] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3149.469300] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3155.542079] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3164.094106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3164.439449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3171.966674] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3181.324236] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3185.012782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3199.990133] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3199.995206] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3200.008589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3200.496862] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3200.515079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3200.524527] Lustre: Skipped 1 previous similar message [ 3205.622074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3206.027712] Lustre: server umount lustre-OST0000 complete [ 3210.720699] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3210.747231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3210.761866] Lustre: Skipped 2 previous similar messages [ 3215.841262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3215.852662] Lustre: Skipped 1 previous similar message [ 3217.050584] Lustre: server umount lustre-MDT0000 complete [ 3220.963609] LustreError: 92462:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3220.977327] LustreError: 92462:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3221.239331] LustreError: 92469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781275349 with bad export cookie 6444405003108396723 [ 3221.252967] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3221.255706] LustreError: 92469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3221.733788] Lustre: server umount lustre-MDT0001 complete [ 3234.907991] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 3238.731374] Key type lgssc unregistered [ 3239.174632] LNet: 95288:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3239.180936] LNetError: 95288:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3239.202775] LNet: Removed LNI 192.168.201.129@tcp [ 3240.408076] Key type .llcrypt unregistered [ 3240.413477] Key type ._llcrypt unregistered [ 3270.544960] Key type ._llcrypt registered [ 3270.557425] Key type .llcrypt registered [ 3270.753926] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 3271.957497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3272.161253] alg: No test for adler32 (adler32-zlib) [ 3273.216356] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 3273.378597] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 3275.031205] Key type lgssc registered [ 3277.137468] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3286.068449] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3286.094731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3301.663952] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3306.784368] LustreError: 96676:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.129@tcp: failed processing log, type 4: rc = -110 [ 3333.471344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3339.104821] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3348.684450] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3356.614705] Lustre: server umount lustre-OST0000 complete [ 3367.722719] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 10:44:54 (1781275494) [ 3377.703446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3378.342194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3383.549952] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3383.789816] LustreError: 98315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3383.812887] LustreError: 98315:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3388.906354] LustreError: 98316:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3392.724115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3393.168638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3398.276462] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3407.884425] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3413.938441] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3420.869876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3421.226397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3424.315146] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3427.995885] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3436.425084] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3448.805610] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3448.809255] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3448.837177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3448.839775] Lustre: Skipped 1 previous similar message [ 3453.925360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3453.947404] Lustre: Skipped 1 previous similar message [ 3455.021333] Lustre: server umount lustre-OST0000 complete [ 3459.552383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3459.563052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3459.580685] Lustre: Skipped 1 previous similar message [ 3459.585753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3464.160918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3464.161259] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3465.700682] Lustre: server umount lustre-MDT0000 complete [ 3469.286921] LustreError: 98315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3469.297925] LustreError: 98315:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3469.623853] LustreError: 98296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781275597 with bad export cookie 6566467154153927158 [ 3469.636627] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3469.652820] LustreError: 98296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3470.223066] Lustre: server umount lustre-MDT0001 complete [ 3482.470517] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 3485.880735] Key type lgssc unregistered [ 3486.314354] LNet: 102007:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3486.328439] LNetError: 102007:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3486.349398] LNet: Removed LNI 192.168.201.129@tcp [ 3487.443724] Key type .llcrypt unregistered [ 3487.447603] Key type ._llcrypt unregistered [ 3534.460547] Key type ._llcrypt registered [ 3534.463394] Key type .llcrypt registered [ 3534.590263] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 3536.709554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3536.744269] alg: No test for adler32 (adler32-zlib) [ 3538.075676] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 3538.349696] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 3540.127811] Key type lgssc registered [ 3541.649507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3550.693572] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3550.720599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3552.103847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3556.364525] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3562.464822] LustreError: 104199:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3562.470584] LustreError: 104199:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3565.842434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3566.184492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3570.578157] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3578.700576] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3585.509638] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3593.581525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3594.069373] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3597.135423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3602.227683] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3611.103349] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3621.858243] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3621.863787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3621.866160] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3626.977112] LustreError: 106023:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3626.995914] LustreError: 106023:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3627.061324] Lustre: server umount lustre-OST0000 complete [ 3632.101696] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3632.113084] Lustre: Skipped 2 previous similar messages [ 3632.116533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3632.129208] Lustre: Skipped 1 previous similar message [ 3637.217639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3637.239275] Lustre: Skipped 1 previous similar message [ 3638.269299] Lustre: server umount lustre-MDT0000 complete [ 3642.342355] LustreError: 105207:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3642.376973] LustreError: 105207:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3643.481863] LustreError: 104179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781275771 with bad export cookie 12807405279488392479 [ 3643.491535] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3643.498039] LustreError: 104179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3644.158898] Lustre: server umount lustre-MDT0001 complete [ 3657.755702] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 3660.703541] Key type lgssc unregistered [ 3660.992121] LNet: 107735:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3660.995305] LNetError: 107735:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3662.053352] LNet: Removed LNI 192.168.201.129@tcp [ 3662.883987] Key type .llcrypt unregistered [ 3662.885865] Key type ._llcrypt unregistered [ 3685.705941] Key type ._llcrypt registered [ 3685.709763] Key type .llcrypt registered [ 3685.884268] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 3687.135966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3687.154269] alg: No test for adler32 (adler32-zlib) [ 3688.305233] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 3688.507099] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 3690.175705] Key type lgssc registered [ 3691.724506] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3701.715729] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3701.738116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3717.407444] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3722.528085] LustreError: 108980:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.129@tcp: failed processing log, type 4: rc = -110 [ 3749.215666] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3756.871090] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3766.395185] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3775.369884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3775.803856] LustreError: MGC192.168.201.129@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3775.816363] LustreError: 110004:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3775.837616] LustreError: 110004:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 3775.844556] LustreError: 110004:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 3776.014733] Lustre: server umount lustre-MDT0000 complete [ 3776.021042] LustreError: 110004:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3793.375206] Lustre: 108317:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781275905/real 1781275905] req@000000002a157312 x1867803070041472/t0(0) o400->MGC192.168.201.129@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781275921 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3793.390445] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3793.397228] LustreError: 108992:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3793.630558] Lustre: server umount lustre-OST0000 complete [ 3805.049953] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_hostid [ 3815.730445] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 3859.713358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3860.086906] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3860.149266] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3860.239376] Lustre: lustre-MDT0000: new disk, initializing [ 3860.361238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3860.381120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3865.079852] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3876.801549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3876.923499] Lustre: 112797:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3876.979341] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3876.987933] Lustre: Skipped 1 previous similar message [ 3877.067606] Lustre: lustre-MDT0001: new disk, initializing [ 3877.184561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3877.224711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3877.233545] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3882.600477] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3892.529085] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3898.774217] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3906.798454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3907.213502] Lustre: lustre-OST0000: new disk, initializing [ 3907.226125] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3907.243636] Lustre: 114768:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3907.383475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3913.270317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3913.289046] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3913.448944] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3914.689751] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 3924.699944] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3932.298874] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3932.632339] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3941.012876] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3941.334056] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3944.484564] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3944.532607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3949.557381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3949.572106] Lustre: Skipped 1 previous similar message [ 3950.773721] Lustre: server umount lustre-OST0000 complete [ 3959.264432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3959.272239] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3959.285360] Lustre: Skipped 1 previous similar message [ 3959.295248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3959.302743] Lustre: Skipped 1 previous similar message [ 3963.674571] Lustre: server umount lustre-MDT0000 complete [ 3968.200543] LustreError: 112790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781276096 with bad export cookie 16429586156025041631 [ 3968.202822] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3968.212279] LustreError: 112790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3968.620834] Lustre: server umount lustre-MDT0001 complete [ 3980.394715] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 10:55:07 (1781276107) [ 3984.614341] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 4005.230800] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_hostid [ 4017.267823] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 4056.515965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4056.844564] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4056.887661] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4056.960492] Lustre: lustre-MDT0000: new disk, initializing [ 4057.038729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4057.049965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4062.159860] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4073.598799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4073.776630] Lustre: 119671:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4073.857777] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4073.860870] Lustre: Skipped 1 previous similar message [ 4073.988895] Lustre: lustre-MDT0001: new disk, initializing [ 4074.195682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4074.269500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4074.283106] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4079.349247] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4090.122214] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4096.754710] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4104.430747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4104.910103] Lustre: lustre-OST0000: new disk, initializing [ 4104.935828] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4104.968988] Lustre: 121639:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4105.081579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4111.634502] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4115.021349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4115.038605] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4115.166678] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4121.828158] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4127.070796] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4127.332708] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4132.444152] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4132.763910] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4135.904870] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4135.905511] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4135.920365] LustreError: Skipped 1 previous similar message [ 4135.921371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4135.921376] Lustre: Skipped 1 previous similar message [ 4135.978313] Lustre: Skipped 2 previous similar messages [ 4141.031909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4141.054488] Lustre: Skipped 2 previous similar messages [ 4141.453464] Lustre: server umount lustre-OST0000 complete [ 4146.144425] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4146.148218] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4146.176686] Lustre: Skipped 1 previous similar message [ 4151.265637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4151.279521] Lustre: Skipped 3 previous similar messages [ 4152.593910] Lustre: server umount lustre-MDT0000 complete [ 4156.179757] LustreError: 121638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781276284 with bad export cookie 16429586156025042513 [ 4156.190316] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4156.198840] LustreError: 121638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4156.438091] Lustre: server umount lustre-MDT0001 complete [ 4166.982081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4167.321847] LustreError: 123857:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4167.344374] LustreError: 123857:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4167.393881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4171.893237] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4172.772993] LustreError: 123858:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4177.903258] LustreError: 123857:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4180.306471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4184.867379] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4195.786280] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4204.099785] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4211.844247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4219.403661] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4226.944125] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4233.314462] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4239.329661] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4239.344786] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4239.373494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4243.022448] Lustre: server umount lustre-OST0000 complete [ 4251.634840] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4251.662457] Lustre: Skipped 2 previous similar messages [ 4253.776514] Lustre: server umount lustre-MDT0000 complete [ 4256.740043] LustreError: 123858:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4256.762953] LustreError: 123858:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4258.289819] LustreError: 123837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781276386 with bad export cookie 16429586156025043395 [ 4258.291918] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4258.298680] LustreError: 123837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4258.636836] Lustre: server umount lustre-MDT0001 complete [ 4270.942869] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 4274.171427] Key type lgssc unregistered [ 4274.459949] LNet: 127487:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4274.482406] LNetError: 127487:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4274.508191] LNet: Removed LNI 192.168.201.129@tcp [ 4275.787603] Key type .llcrypt unregistered [ 4275.793243] Key type ._llcrypt unregistered [ 4298.317812] Key type ._llcrypt registered [ 4298.322532] Key type .llcrypt registered [ 4298.465724] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_hostid [ 4318.882898] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 4320.609272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4320.653022] alg: No test for adler32 (adler32-zlib) [ 4321.694156] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 4321.945782] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 4323.648108] Key type lgssc registered [ 4325.184614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4365.806992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4365.890566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4367.238993] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4367.270397] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4367.341773] Lustre: lustre-MDT0000: new disk, initializing [ 4367.402559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4367.415976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4372.177298] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4384.240915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4384.386622] Lustre: 130654:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4384.438773] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4384.450970] Lustre: Skipped 1 previous similar message [ 4384.567485] Lustre: lustre-MDT0001: new disk, initializing [ 4384.716294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4384.764920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4384.786499] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4389.099122] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4399.262950] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4406.355849] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4414.792053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4415.215128] Lustre: lustre-OST0000: new disk, initializing [ 4415.217936] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4415.223631] Lustre: 132622:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4415.301758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4421.168806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4421.174680] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4421.301225] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4423.007252] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4432.722953] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4438.755438] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4438.935882] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4446.234453] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4446.507960] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4452.325487] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4452.326101] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4452.338014] LustreError: Skipped 1 previous similar message [ 4452.352687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4454.851848] Lustre: server umount lustre-OST0000 complete [ 4461.542875] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4461.550680] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4461.563756] Lustre: Skipped 1 previous similar message [ 4461.567030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4461.575855] Lustre: Skipped 1 previous similar message [ 4462.568651] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4462.602561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4462.617576] Lustre: Skipped 1 previous similar message [ 4465.756588] Lustre: server umount lustre-MDT0000 complete [ 4470.364406] LustreError: 130645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781276598 with bad export cookie 1363220957695334706 [ 4470.389279] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4470.815260] Lustre: server umount lustre-MDT0001 complete [ 4484.601585] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 11:03:30 (1781276610) [ 4492.952368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4493.289629] LustreError: 135036:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4493.301531] LustreError: 135036:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4493.353723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4498.409818] LustreError: 135037:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4499.312278] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4503.537103] LustreError: 135036:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4508.642442] LustreError: 135037:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4509.286220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4509.798082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4514.401968] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4522.231634] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4530.106800] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4534.240903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4534.283590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4534.295851] Lustre: Skipped 1 previous similar message [ 4539.237709] Lustre: server umount lustre-MDT0000 complete [ 4540.387457] LustreError: 135036:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4540.419451] LustreError: 135036:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4543.683016] LustreError: 135018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781276671 with bad export cookie 1363220957695335588 [ 4543.695801] LustreError: 135018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4543.717781] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4544.119549] Lustre: server umount lustre-MDT0001 complete [ 4554.542131] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 11:04:41 (1781276681) [ 4563.462072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4579.103560] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4584.223490] LustreError: 137453:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.129@tcp: failed processing log, type 4: rc = -110 [ 4609.951334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4616.193435] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4624.988635] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4627.701105] Lustre: server umount lustre-OST0000 complete [ 4636.544740] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 11:06:03 (1781276763) [ 4644.049598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4644.585616] LustreError: 138900:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4644.667685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4649.805119] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4659.346790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4665.690097] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4675.093785] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4681.809592] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4690.064204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4690.305539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4690.310080] Lustre: Skipped 1 previous similar message [ 4698.225277] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4707.234771] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4724.192783] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4724.198127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4724.200835] Lustre: Skipped 3 previous similar messages [ 4724.209156] Lustre: Skipped 4 previous similar messages [ 4728.958923] Lustre: server umount lustre-MDT0000 complete [ 4729.319877] LustreError: 140977:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4729.331503] LustreError: 140977:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4734.007933] LustreError: 140746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781276862 with bad export cookie 1363220957695336253 [ 4734.010184] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4734.022423] LustreError: 140746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4734.539982] Lustre: server umount lustre-MDT0001 complete [ 4748.230384] Lustre: server umount lustre-OST0000 complete [ 4757.848536] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 11:08:04 (1781276884) [ 4764.825972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4765.274731] LustreError: 142576:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4765.345815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4770.971395] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4779.601405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4785.248053] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4794.778829] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4803.106106] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4811.949757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4819.566415] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4830.109356] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4835.020914] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4835.586722] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4835.766720] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4840.859866] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4841.054864] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4845.539577] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4845.549453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4845.570768] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4849.666955] Lustre: server umount lustre-OST0000 complete [ 4859.838922] Lustre: server umount lustre-MDT0000 complete [ 4860.896893] LustreError: 142576:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4860.913597] LustreError: 142576:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4863.446269] LustreError: 142556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781276991 with bad export cookie 1363220957695337324 [ 4863.458344] LustreError: 142556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4863.459404] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4863.828877] Lustre: server umount lustre-MDT0001 complete [ 4873.567962] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 11:10:00 (1781277000) [ 4879.888189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4895.455675] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4900.575545] LustreError: 146738:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.129@tcp: failed processing log, type 4: rc = -110 [ 4926.303914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4926.311506] Lustre: Skipped 2 previous similar messages [ 4933.247432] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4941.857787] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4950.764742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4951.345624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4957.361750] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4966.185264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4971.632778] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 4980.779463] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4987.385334] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4993.051557] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4993.306234] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4998.729195] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4998.883341] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5002.730786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5002.736280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5002.739726] Lustre: Skipped 4 previous similar messages [ 5002.739915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5002.739919] Lustre: Skipped 3 previous similar messages [ 5007.841975] LustreError: 147777:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5007.859081] LustreError: 147777:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5007.875743] Lustre: server umount lustre-MDT0000 complete [ 5015.148777] LustreError: 146744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781277143 with bad export cookie 1363220957695338346 [ 5015.155324] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5015.675827] Lustre: server umount lustre-MDT0001 complete [ 5031.681272] Lustre: server umount lustre-OST0000 complete [ 5041.786558] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 11:12:48 (1781277168) [ 5048.049978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5063.714217] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5068.896379] LustreError: 150936:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.129@tcp: failed processing log, type 4: rc = -110 [ 5102.191746] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 5111.454728] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5119.557828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5120.213813] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 5125.562669] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 5135.390751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5140.926727] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 5148.771739] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5155.089359] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5163.122142] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5163.315903] Lustre: lustre-OST0001: new disk, initializing [ 5163.319213] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5163.325622] Lustre: 153811:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5169.890446] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 5172.301627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5172.320368] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5172.481765] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5180.274427] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5184.927581] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5185.072829] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5189.312582] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5189.508835] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5191.648928] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5191.651698] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5191.663729] Lustre: Skipped 1 previous similar message [ 5191.672505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5191.674534] Lustre: Skipped 2 previous similar messages [ 5197.566768] Lustre: server umount lustre-OST0000 complete [ 5203.437970] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5203.448293] LustreError: Skipped 1 previous similar message [ 5207.800086] Lustre: server umount lustre-OST0001 complete [ 5218.309470] Lustre: server umount lustre-MDT0000 complete [ 5222.380992] LustreError: 150944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781277350 with bad export cookie 1363220957695339298 [ 5222.392583] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5222.399126] LustreError: 150944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5223.003756] Lustre: server umount lustre-MDT0001 complete [ 5248.464931] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 11:16:15 (1781277375) [ 5250.032282] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5251.862406] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5253.143124] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 11:16:20 (1781277380) [ 5259.786059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5260.039070] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5260.087919] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5260.252566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5260.257081] Lustre: Skipped 6 previous similar messages [ 5264.484783] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 5272.532097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5272.633735] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5272.651935] Lustre: 157375:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5276.877452] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 5284.513138] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5290.727900] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5297.004289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5297.257501] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5302.936603] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 5304.917923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5309.530862] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5313.415625] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5313.585701] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5317.938290] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5318.185536] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5320.160183] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5320.163703] LustreError: Skipped 2 previous similar messages [ 5320.167709] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5320.174881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5320.186409] Lustre: Skipped 6 previous similar messages [ 5320.201828] Lustre: Skipped 8 previous similar messages [ 5326.412839] Lustre: server umount lustre-OST0000 complete [ 5349.616762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5350.953098] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5350.967101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5350.968069] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5350.999997] Lustre: Skipped 1 previous similar message [ 5355.571993] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 5364.479494] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5370.415833] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5370.687909] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5376.403317] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5376.700571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5384.162829] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff893cc9358800.ost_server_uuid 50 [ 5386.533453] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff893cc9358800.ost_server_uuid in FULL state after 0 sec [ 5386.750184] LustreError: 157383:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.29@tcp) returned error from blocking AST (req@00000000db057a14 x1867803735108352 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000096e977ad/0x12eb223d05cc747c lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.29@tcp remote: 0x8f713c922c597bd0 expref: 5 pid: 157381 timeout: 5486 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5386.784973] LustreError: lustre-MDT0000: A client on nid 192.168.201.29@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5386.799443] LustreError: 157371:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.29@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000096e977ad/0x12eb223d05cc747c lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.29@tcp remote: 0x8f713c922c597bd0 expref: 6 pid: 157381 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5396.959954] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5406.175154] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5406.350762] Lustre: server umount lustre-OST0000 complete [ 5416.658767] Lustre: server umount lustre-MDT0000 complete [ 5417.441586] LustreError: 157383:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5417.464541] LustreError: 157383:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [ 5420.854326] LustreError: 157368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781277548 with bad export cookie 1363220957695340481 [ 5420.867017] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5420.871728] LustreError: 157368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5435.359160] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5435.759656] Lustre: server umount lustre-MDT0001 complete [ 5446.701661] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 5449.006606] Key type lgssc unregistered [ 5449.252640] LNet: 163488:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5449.271757] LNetError: 163488:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5449.295452] LNet: Removed LNI 192.168.201.129@tcp [ 5449.807956] Key type .llcrypt unregistered [ 5449.809193] Key type ._llcrypt unregistered [ 5475.425745] Key type ._llcrypt registered [ 5475.430669] Key type .llcrypt registered [ 5475.662976] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 5476.777932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5477.094587] alg: No test for adler32 (adler32-zlib) [ 5478.296556] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 5478.579562] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 5480.240162] Key type lgssc registered [ 5481.442302] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5489.444945] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5489.476397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5491.113541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5495.971568] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 5501.416145] LustreError: 164908:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5501.432996] LustreError: 164908:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5504.356733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5504.937062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5510.831604] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 5520.826889] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5528.162622] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5535.695373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5536.205531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5537.232602] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5542.812817] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 5551.162469] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5558.363842] Lustre: Failing over lustre-MDT0000 [ 5558.642594] Lustre: server umount lustre-MDT0000 complete [ 5559.064621] LustreError: 166071:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5559.780322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5559.807204] Lustre: Skipped 2 previous similar messages [ 5564.897485] LustreError: 165636:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5564.919918] LustreError: 165636:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5570.023281] LustreError: 164903:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5570.050662] LustreError: 164903:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5571.070742] LustreError: 164890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.29@tcp arrived at 1781277699 with bad export cookie 7083259774728530080 [ 5571.080792] LustreError: 164890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5575.136720] LustreError: 164909:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5575.137246] Lustre: 164219:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781277687/real 1781277687] req@0000000074219721 x1867804947018112/t0(0) o400->MGC192.168.201.129@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781277703 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5575.152852] LustreError: 164909:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5575.194151] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5577.447115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5585.377845] LustreError: 164905:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5585.399222] LustreError: 164905:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5586.403663] LustreError: 164217:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001ea45e16 x1867804947023360/t0(0) o250->MGC192.168.201.129@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5586.719730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5586.789889] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5591.041389] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 5592.036586] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5592.061364] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5593.497662] Lustre: lustre-MDT0000: Denying connection for new client 7e22772a-7e1e-4aef-9bce-128025b0eedc (at 192.168.201.29@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 5605.857825] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5605.861138] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5605.893982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5607.399780] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5607.401951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5607.427054] Lustre: Skipped 1 previous similar message [ 5608.642232] Lustre: server umount lustre-OST0000 complete [ 5612.730355] LustreError: 168641:0:(ldlm_lib.c:2987:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5612.739271] Lustre: 167991:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5612.745132] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5612.763826] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5612.777829] LustreError: 167991:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000096b55cc5 x1867804947038848/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5612.791507] LustreError: 167991:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5612.804186] LustreError: 167991:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5612.822168] Lustre: lustre-MDT0000: Recovery over after 0:20, of 2 clients 0 recovered and 2 were evicted. [ 5613.314714] Lustre: server umount lustre-MDT0000 complete [ 5616.903068] LustreError: 164888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781277745 with bad export cookie 7083259774728530178 [ 5616.915367] LustreError: 164888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5616.921916] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5616.926902] LustreError: 164217:0:(import.c:709:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5623.328157] Lustre: server umount lustre-MDT0001 complete [ 5633.736996] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 5636.644947] Key type lgssc unregistered [ 5636.990767] LNet: 169393:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5636.997851] LNetError: 169393:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5637.007916] LNet: Removed LNI 192.168.201.129@tcp [ 5637.772089] Key type .llcrypt unregistered [ 5637.773284] Key type ._llcrypt unregistered [ 5659.182949] Key type ._llcrypt registered [ 5659.184633] Key type .llcrypt registered [ 5659.233476] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 5659.982418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5660.059257] alg: No test for adler32 (adler32-zlib) [ 5661.090806] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 5661.282454] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 5662.967186] Key type lgssc registered [ 5664.385525] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5674.088674] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5674.119331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5675.655667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5680.757995] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 5685.232559] LustreError: 170783:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5685.259712] LustreError: 170783:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5690.336833] LustreError: 170784:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5691.023649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5691.497053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5697.008682] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 5704.974167] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5710.607873] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5716.892681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5717.340312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5720.436860] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5723.428156] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 5732.514849] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5741.025342] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5741.035482] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5741.057091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5742.049253] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5742.062151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5742.068604] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5742.086422] Lustre: Skipped 1 previous similar message [ 5746.568839] Lustre: server umount lustre-OST0000 complete [ 5752.289034] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5752.303034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5752.321143] Lustre: Skipped 1 previous similar message [ 5757.417081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5757.426393] Lustre: Skipped 1 previous similar message [ 5758.260602] Lustre: server umount lustre-MDT0000 complete [ 5762.145342] LustreError: 170765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781277890 with bad export cookie 18422730018488256367 [ 5762.154871] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5762.158227] LustreError: 170765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5762.599491] Lustre: server umount lustre-MDT0001 complete [ 5776.654312] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 5779.733841] Key type lgssc unregistered [ 5780.169792] LNet: 174318:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5780.177659] LNetError: 174318:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5780.213601] LNet: Removed LNI 192.168.201.129@tcp [ 5781.326114] Key type .llcrypt unregistered [ 5781.328499] Key type ._llcrypt unregistered [ 5816.753504] Key type ._llcrypt registered [ 5816.755686] Key type .llcrypt registered [ 5816.958979] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 5818.522987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5818.585161] alg: No test for adler32 (adler32-zlib) [ 5819.682595] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 5819.903883] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 5821.663802] Key type lgssc registered [ 5822.825901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5831.550539] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5831.594868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5833.136501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5838.326289] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 5843.424733] LustreError: 176024:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5843.437789] LustreError: 176024:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5847.174589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5847.511891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5852.406217] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 5860.862584] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5866.706691] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5873.270480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5873.600481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5878.719808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5879.584881] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 5887.130872] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5897.184460] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5897.191316] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5897.220333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5899.228817] Lustre: server umount lustre-OST0000 complete [ 5903.851699] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5903.859136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5903.881971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5905.376924] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5905.382506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5905.412605] Lustre: Skipped 1 previous similar message [ 5910.499094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5910.507127] Lustre: Skipped 1 previous similar message [ 5913.720848] Lustre: server umount lustre-MDT0000 complete [ 5915.618884] LustreError: 176019:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5915.638673] LustreError: 176019:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5917.124684] LustreError: 176005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781278045 with bad export cookie 17295156633703837989 [ 5917.138400] LustreError: 176005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5917.142949] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5917.496399] Lustre: server umount lustre-MDT0001 complete [ 5930.933717] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 5935.463989] Key type lgssc unregistered [ 5935.992032] LNet: 179557:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5936.021421] LNetError: 179557:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5936.053799] LNet: Removed LNI 192.168.201.129@tcp [ 5937.024306] Key type .llcrypt unregistered [ 5937.029787] Key type ._llcrypt unregistered [ 5963.703862] Key type ._llcrypt registered [ 5963.705231] Key type .llcrypt registered [ 5963.898231] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 5965.978161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5966.095537] alg: No test for adler32 (adler32-zlib) [ 5967.978563] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 5968.416710] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 5970.127172] Key type lgssc registered [ 5972.402780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5988.167640] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5988.223327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5990.069771] Lustre: *** cfs_fail_loc=135, val=0*** [ 5990.238184] LustreError: 181100:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5990.242537] LustreError: 181100:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.129@tcp: cfg command failed: rc = -2 [ 5990.249325] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5990.249325] [ 5990.264703] LustreError: MGC192.168.201.129@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5990.271132] LustreError: 181061:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5990.274254] LustreError: 181061:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 5990.276989] LustreError: 181061:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 5990.279753] LustreError: 181061:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5990.397710] Lustre: server umount lustre-MDT0000 complete [ 5990.399662] LustreError: 181061:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5998.503039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5999.041668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6004.025792] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 6009.320076] LustreError: 181472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6009.353153] LustreError: 181472:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6013.731664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6014.207617] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6019.252858] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 6027.502845] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6033.325331] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6039.778278] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6039.780373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6039.786433] Lustre: Skipped 1 previous similar message [ 6039.795955] Lustre: Skipped 1 previous similar message [ 6042.010242] Lustre: server umount lustre-MDT0000 complete [ 6045.162132] LustreError: 181471:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6045.173945] LustreError: 181471:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6046.366644] LustreError: 182206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781278174 with bad export cookie 111039526547147819 [ 6046.376383] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6046.384083] LustreError: 182206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6046.773258] Lustre: server umount lustre-MDT0001 complete [ 6060.981267] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 6064.430534] Key type lgssc unregistered [ 6064.849914] LNet: 183793:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6064.863543] LNetError: 183793:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6064.914858] LNet: Removed LNI 192.168.201.129@tcp [ 6065.893280] Key type .llcrypt unregistered [ 6065.895794] Key type ._llcrypt unregistered [ 6092.315686] Key type ._llcrypt registered [ 6092.318335] Key type .llcrypt registered [ 6092.449079] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 6092.773906] Key type .llcrypt unregistered [ 6092.784222] Key type ._llcrypt unregistered [ 6110.652641] Key type ._llcrypt registered [ 6110.656239] Key type .llcrypt registered [ 6110.783803] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 6111.764963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6111.903133] alg: No test for adler32 (adler32-zlib) [ 6113.059479] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 6113.349913] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 6115.039610] Key type lgssc registered [ 6116.193304] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6123.280469] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6123.326181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6138.911355] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6144.034242] LustreError: 185933:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.129@tcp: failed processing log, type 4: rc = -110 [ 6170.655179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6179.995070] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 6188.439793] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6196.162578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6196.850682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6196.922805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6201.838530] LustreError: 185960:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6201.850756] LustreError: 185960:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6202.473805] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 6204.907853] LustreError: 186972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6204.929483] LustreError: 186972:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6210.040852] LustreError: 185959:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6210.057469] LustreError: 185959:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6211.650644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6212.334181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6218.473983] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 6228.084688] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6234.698597] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6237.707427] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6247.904343] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6247.915801] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6247.933628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6248.416568] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6248.432135] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6248.446050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6248.462614] Lustre: Skipped 1 previous similar message [ 6253.540901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6253.548604] Lustre: Skipped 1 previous similar message [ 6253.952556] Lustre: server umount lustre-OST0000 complete [ 6258.657543] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6258.662800] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6258.670657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6258.691236] Lustre: Skipped 1 previous similar message [ 6263.777512] LustreError: 186976:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6263.781579] Lustre: server umount lustre-MDT0000 complete [ 6267.428246] LustreError: 185941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781278395 with bad export cookie 14547919011440173344 [ 6267.432837] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6267.437259] LustreError: 185941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6268.172774] Lustre: server umount lustre-MDT0001 complete [ 6280.605892] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 6284.091730] Key type lgssc unregistered [ 6284.363801] LNet: 189854:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6284.383621] LNetError: 189854:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6284.398776] LNet: Removed LNI 192.168.201.129@tcp [ 6285.253322] Key type .llcrypt unregistered [ 6285.257379] Key type ._llcrypt unregistered [ 6310.816700] Key type ._llcrypt registered [ 6310.819659] Key type .llcrypt registered [ 6310.921234] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 6312.077042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6312.289930] alg: No test for adler32 (adler32-zlib) [ 6313.379603] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 6313.621919] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 6315.303403] Key type lgssc registered [ 6316.536183] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6324.906704] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6324.920518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6326.441869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6330.582915] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 6335.972108] LustreError: 191269:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6335.996477] LustreError: 191269:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6339.779494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6340.076889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6344.403351] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 6352.798388] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6359.140077] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6367.016736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6367.418314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6371.519925] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6374.346561] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 6383.044507] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6389.365246] Lustre: Failing over lustre-MDT0000 [ 6389.676421] Lustre: server umount lustre-MDT0000 complete [ 6391.069357] LustreError: 192811:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6391.273648] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6391.277616] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6391.285767] Lustre: Skipped 1 previous similar message [ 6396.150888] LustreError: 193714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6396.164266] LustreError: 193714:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6401.269763] LustreError: 191264:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6401.282617] LustreError: 191264:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6406.380928] LustreError: 193713:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6406.392766] LustreError: 193713:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6407.647259] Lustre: 190581:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781278519/real 1781278519] req@0000000069670acd x1867805822576768/t0(0) o400->MGC192.168.201.129@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781278535 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6407.698858] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6408.906807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6416.622148] LustreError: 192811:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6416.658938] LustreError: 192811:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 6417.156208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6417.181912] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6421.335618] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 6421.741658] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6422.520087] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6422.539046] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 6422.613146] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6426.445833] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6428.467383] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6437.344132] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6437.351418] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6437.360682] Lustre: Skipped 1 previous similar message [ 6437.364431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6437.864789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6442.984114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6443.554944] Lustre: server umount lustre-OST0000 complete [ 6448.100519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6448.104640] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6448.113179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6448.119261] Lustre: Skipped 2 previous similar messages [ 6448.129528] Lustre: Skipped 2 previous similar messages [ 6453.216361] LustreError: 191268:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6453.222367] LustreError: 191268:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6453.247919] Lustre: server umount lustre-MDT0000 complete [ 6456.427787] LustreError: 191250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781278584 with bad export cookie 10414553297821543816 [ 6456.433424] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6456.454627] LustreError: 191250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6456.705374] Lustre: server umount lustre-MDT0001 complete [ 6468.417881] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 6471.045318] Key type lgssc unregistered [ 6471.484185] LNet: 196008:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6471.487607] LNetError: 196008:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6471.520504] LNet: Removed LNI 192.168.201.129@tcp [ 6472.629841] Key type .llcrypt unregistered [ 6472.633641] Key type ._llcrypt unregistered [ 6501.828202] Key type ._llcrypt registered [ 6501.829837] Key type .llcrypt registered [ 6502.002385] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 6503.704665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6503.746483] alg: No test for adler32 (adler32-zlib) [ 6504.876523] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 6505.049040] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 6506.711206] Key type lgssc registered [ 6508.004708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6515.175382] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6515.206187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6516.810175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6522.070937] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 6525.930916] LustreError: 197423:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6525.957718] LustreError: 197423:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6531.042763] LustreError: 197422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6533.669797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6534.283277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6539.729770] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 6549.075799] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6555.373832] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6562.413951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6562.866180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6568.203815] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 6577.416381] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6586.368703] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6592.051069] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6594.558250] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6611.683947] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6626.272559] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6626.289410] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6626.304353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6627.298555] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6627.299260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6631.284382] Lustre: server umount lustre-OST0000 complete [ 6636.512837] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6636.518950] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6636.567902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6636.573437] Lustre: Skipped 1 previous similar message [ 6642.657246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6642.659519] Lustre: Skipped 3 previous similar messages [ 6644.898245] Lustre: server umount lustre-MDT0000 complete [ 6647.781869] LustreError: 198153:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6647.793813] LustreError: 198153:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6648.718015] LustreError: 197402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781278776 with bad export cookie 18397761343255242404 [ 6648.725801] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6648.727385] LustreError: 197402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6649.019228] Lustre: server umount lustre-MDT0001 complete [ 6657.647332] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 6660.277865] Key type lgssc unregistered [ 6660.668773] LNet: 201337:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6660.687137] LNetError: 201337:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6660.710334] LNet: Removed LNI 192.168.201.129@tcp [ 6661.916876] Key type .llcrypt unregistered [ 6661.920342] Key type ._llcrypt unregistered [ 6682.703240] Key type ._llcrypt registered [ 6682.705481] Key type .llcrypt registered [ 6682.833869] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 6684.222171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6684.243206] alg: No test for adler32 (adler32-zlib) [ 6685.528366] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 6685.755772] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 6687.479242] Key type lgssc registered [ 6688.910428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6697.405290] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6697.437558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6698.834425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6703.687527] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 6708.192910] LustreError: 202605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6708.213397] LustreError: 202605:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6712.121234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6712.547779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6717.535884] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 6725.990789] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6732.464453] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6739.731200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6740.060564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6743.159788] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6746.974980] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 6755.244914] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6762.154778] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6772.211410] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6778.742894] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6783.978102] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6793.700636] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6793.708499] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6793.727326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6793.736490] Lustre: Skipped 1 previous similar message [ 6798.821401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6803.941558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6803.948504] Lustre: Skipped 2 previous similar messages [ 6805.983317] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6806.187731] Lustre: server umount lustre-OST0000 complete [ 6810.088588] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6810.091549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6810.106598] Lustre: Skipped 1 previous similar message [ 6810.114388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6814.176607] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6814.176781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6814.195634] Lustre: Skipped 1 previous similar message [ 6816.529202] Lustre: server umount lustre-MDT0000 complete [ 6819.296296] LustreError: 204863:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6819.317115] LustreError: 204863:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6819.858127] LustreError: 202585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781278947 with bad export cookie 8091707909089233915 [ 6819.880415] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6819.884359] LustreError: 202585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6820.151834] Lustre: server umount lustre-MDT0001 complete [ 6834.969917] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 6838.260372] Key type lgssc unregistered [ 6838.727051] LNet: 207386:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6838.753552] LNetError: 207386:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6838.784762] LNet: Removed LNI 192.168.201.129@tcp [ 6840.225179] Key type .llcrypt unregistered [ 6840.231063] Key type ._llcrypt unregistered [ 6867.761649] Key type ._llcrypt registered [ 6867.763435] Key type .llcrypt registered [ 6867.908590] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 6868.922629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6869.036338] alg: No test for adler32 (adler32-zlib) [ 6870.131876] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 6870.331672] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 6872.047159] Key type lgssc registered [ 6873.058302] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6881.029875] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6881.059519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6882.599838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6886.903475] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 6892.004446] LustreError: 208800:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6892.018260] LustreError: 208800:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6894.637339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6894.881604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6898.752455] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 6904.673208] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6910.026410] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6916.043595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6916.656757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6923.931221] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 6926.916573] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6931.406552] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6947.300272] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6947.301840] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6947.304889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6947.322277] Lustre: Skipped 1 previous similar message [ 6952.422029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6952.427833] Lustre: Skipped 1 previous similar message [ 6953.468532] Lustre: server umount lustre-OST0000 complete [ 6957.538165] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6957.549309] Lustre: Skipped 1 previous similar message [ 6957.555487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6957.558176] Lustre: Skipped 2 previous similar messages [ 6962.659190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6962.664692] Lustre: Skipped 1 previous similar message [ 6967.777362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6967.786164] Lustre: Skipped 1 previous similar message [ 6971.360262] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6971.667224] Lustre: server umount lustre-MDT0000 complete [ 6972.903696] LustreError: 208800:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6972.921654] LustreError: 208800:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6975.481242] LustreError: 209535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781279103 with bad export cookie 16680590074367005533 [ 6975.483264] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6975.493438] LustreError: 209535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6975.778575] Lustre: server umount lustre-MDT0001 complete [ 6986.591045] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 6989.134929] Key type lgssc unregistered [ 6989.438817] LNet: 212484:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6989.448045] LNetError: 212484:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6990.506324] LNet: Removed LNI 192.168.201.129@tcp [ 6991.567732] Key type .llcrypt unregistered [ 6991.572200] Key type ._llcrypt unregistered [ 7010.324838] Key type ._llcrypt registered [ 7010.327131] Key type .llcrypt registered [ 7010.505728] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 7011.725326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7011.906087] alg: No test for adler32 (adler32-zlib) [ 7012.950465] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 7013.191791] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 7014.903161] Key type lgssc registered [ 7015.941568] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7024.776670] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7024.799235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7026.331704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7030.742879] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 7036.896984] LustreError: 213748:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7036.911203] LustreError: 213748:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7038.687729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7039.123565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7044.008436] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 7051.015616] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7056.104668] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7063.483169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7063.869152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7069.704835] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 7072.048978] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 7077.039412] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7092.711241] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7092.725669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7092.735113] Lustre: Skipped 1 previous similar message [ 7097.477753] Lustre: server umount lustre-OST0000 complete [ 7102.947732] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7102.985672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7103.000214] Lustre: Skipped 2 previous similar messages [ 7108.064815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7108.079742] Lustre: Skipped 1 previous similar message [ 7108.820738] Lustre: server umount lustre-MDT0000 complete [ 7112.166288] LustreError: 213749:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7112.197303] LustreError: 213749:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7112.596422] LustreError: 213728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781279240 with bad export cookie 959838271183003823 [ 7112.615514] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7112.919908] Lustre: server umount lustre-MDT0001 complete [ 7122.212654] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 7125.088875] Key type lgssc unregistered [ 7125.427822] LNet: 217429:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7125.434624] LNetError: 217429:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7125.462793] LNet: Removed LNI 192.168.201.129@tcp [ 7126.404518] Key type .llcrypt unregistered [ 7126.407746] Key type ._llcrypt unregistered [ 7145.822738] Key type ._llcrypt registered [ 7145.824206] Key type .llcrypt registered [ 7145.886705] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 7146.872562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7146.977511] alg: No test for adler32 (adler32-zlib) [ 7147.951067] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 7148.118500] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 7149.807194] Key type lgssc registered [ 7150.913506] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7159.723618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7159.779390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7161.427353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7166.295556] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 7171.558499] LustreError: 218696:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7171.587887] LustreError: 218696:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7176.191668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7176.676497] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7177.001721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7184.182153] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 7197.342468] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7202.854687] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7208.676076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7209.227157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7214.176176] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 7222.912480] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7232.748906] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7244.654565] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7244.695558] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7244.815192] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7253.099920] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 7261.189960] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7262.776292] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7274.899537] Lustre: Permanently deactivating lustre-OST0001 [ 7274.918948] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7280.778488] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7287.613207] Lustre: Permanently reactivating lustre-OST0001 [ 7287.626047] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7287.633537] Lustre: Skipped 2 previous similar messages [ 7295.042750] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7295.056733] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7295.074320] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7295.100192] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7295.145565] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7298.902783] Lustre: server umount lustre-OST0001 complete [ 7300.610324] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7300.613493] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7300.645062] LustreError: Skipped 1 previous similar message [ 7300.667622] Lustre: Skipped 1 previous similar message [ 7300.680846] LustreError: 220519:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7300.720735] LustreError: 220519:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7303.648250] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7303.653625] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7303.659576] Lustre: Skipped 1 previous similar message [ 7303.663659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7304.683155] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7304.691781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7305.709265] LustreError: 220519:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7305.726657] LustreError: 220519:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7309.585234] Lustre: server umount lustre-OST0000 complete [ 7315.427169] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7315.435105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7315.449389] Lustre: Skipped 1 previous similar message [ 7315.459231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7315.475278] Lustre: Skipped 2 previous similar messages [ 7320.836494] Lustre: server umount lustre-MDT0000 complete [ 7321.058229] LustreError: 218690:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7321.092737] LustreError: 218690:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7325.037790] LustreError: 218678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781279453 with bad export cookie 13188587290721176878 [ 7325.062573] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7325.099144] LustreError: 218678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7325.596972] Lustre: server umount lustre-MDT0001 complete [ 7338.544353] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 7341.388410] Key type lgssc unregistered [ 7341.933218] LNet: 223765:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7341.949880] LNetError: 223765:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7343.021244] LNet: Removed LNI 192.168.201.129@tcp [ 7343.740074] Key type .llcrypt unregistered [ 7343.746488] Key type ._llcrypt unregistered [ 7399.225941] Key type ._llcrypt registered [ 7399.228472] Key type .llcrypt registered [ 7399.293085] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 7400.527652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7400.553152] alg: No test for adler32 (adler32-zlib) [ 7401.600695] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 7401.801684] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 7403.488164] Key type lgssc registered [ 7404.522142] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7413.627605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7413.641805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7415.021299] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7415.092249] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7415.354461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7421.021919] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 7432.600995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7432.829377] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7432.921425] Lustre: 225973:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7433.356859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7440.919654] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 7452.388102] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7459.386148] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7467.939763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7468.127576] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7468.304045] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7474.295742] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 7476.910601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7482.432562] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7487.017840] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7489.575646] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7497.735387] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7503.631322] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7513.269587] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7529.609538] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7529.612770] Lustre: Skipped 1 previous similar message [ 7563.616832] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7563.623304] Lustre: Skipped 3 previous similar messages [ 7629.647390] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7629.655939] Lustre: Skipped 8 previous similar messages [ 7661.023960] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7661.033628] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7661.056444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7661.542138] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7661.558099] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7661.590980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7665.583857] Lustre: server umount lustre-OST0000 complete [ 7671.782523] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7671.803737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7671.808863] Lustre: Skipped 1 previous similar message [ 7676.205049] Lustre: server umount lustre-MDT0000 complete [ 7679.925560] LustreError: 225963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781279808 with bad export cookie 15027135799899443216 [ 7679.933130] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7679.940757] LustreError: 225963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7680.244138] Lustre: server umount lustre-MDT0001 complete [ 7692.953958] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 7696.783917] Key type lgssc unregistered [ 7696.976407] LNet: 230763:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7696.986888] LNetError: 230763:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7696.999778] LNet: Removed LNI 192.168.201.129@tcp [ 7697.541511] Key type .llcrypt unregistered [ 7697.543915] Key type ._llcrypt unregistered [ 7724.822401] Key type ._llcrypt registered [ 7724.828871] Key type .llcrypt registered [ 7725.048881] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 7726.424787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7726.588858] alg: No test for adler32 (adler32-zlib) [ 7727.739410] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 7728.021348] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 7729.743595] Key type lgssc registered [ 7731.171227] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7739.752110] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7739.767438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7741.224943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7745.724982] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 7751.648251] LustreError: 232178:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7751.674342] LustreError: 232178:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7754.138148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7754.608530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7758.025332] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 7764.817439] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7769.981766] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7777.122704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7777.587203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7783.839262] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_default_debug -1 all [ 7786.809129] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7791.967559] Lustre: DEBUG MARKER: oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7810.531639] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7810.536692] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7810.545798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7812.577586] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7812.580851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7812.588472] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7817.709590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7817.720140] Lustre: Skipped 2 previous similar messages [ 7822.304902] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7822.657353] Lustre: server umount lustre-OST0000 complete [ 7827.937946] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7827.945312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7827.957836] Lustre: Skipped 1 previous similar message [ 7827.972051] Lustre: Skipped 1 previous similar message [ 7833.057584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7833.064442] Lustre: Skipped 1 previous similar message [ 7835.342793] Lustre: server umount lustre-MDT0000 complete [ 7838.185139] LustreError: 232178:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7838.213563] LustreError: 232178:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7839.219449] LustreError: 234759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781279967 with bad export cookie 17917552286093967082 [ 7839.233277] LustreError: 234759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7839.237209] LustreError: MGC192.168.201.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7839.570118] Lustre: server umount lustre-MDT0001 complete [ 7849.977406] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 7852.430658] Key type lgssc unregistered [ 7852.663787] LNet: 235870:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7852.679032] LNetError: 235870:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7852.696727] LNet: Removed LNI 192.168.201.129@tcp [ 7853.724282] Key type .llcrypt unregistered [ 7853.727270] Key type ._llcrypt unregistered [ 7876.007187] Key type ._llcrypt registered [ 7876.013908] Key type .llcrypt registered [ 7876.110797] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 7877.392720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7877.458089] alg: No test for adler32 (adler32-zlib) [ 7878.444805] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 7878.688957] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 7880.376735] Key type lgssc registered [ 7881.263403] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8151.602870] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 8153.900181] Key type lgssc unregistered [ 8154.132912] LNet: 237903:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8154.142489] LNetError: 237903:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8154.158049] LNet: Removed LNI 192.168.201.129@tcp [ 8154.652473] Key type .llcrypt unregistered [ 8154.655370] Key type ._llcrypt unregistered [ 8181.807669] Key type ._llcrypt registered [ 8181.818257] Key type .llcrypt registered [ 8181.944723] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing set_hostid [ 8198.724980] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing load_modules_local [ 8200.175892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8200.208161] alg: No test for adler32 (adler32-zlib) [ 8201.317797] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 8201.563502] LNet: Added LNI 192.168.201.129@tcp [8/256/0/180] [ 8203.255189] Key type lgssc registered [ 8204.183474] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8240.481142] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7924 sec ========== 12:06:07 (1781280367) [ 8243.006866] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:06:09 (1781280369) === [ 8247.515410] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:06:13 (1781280373) === [ 8302.612131] Lustre: DEBUG MARKER: oleg129-server.virtnet: executing unload_modules_local [ 8306.204101] Key type lgssc unregistered [ 8306.944383] LNet: 243162:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8306.956448] LNetError: 243162:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8306.986492] LNet: Removed LNI 192.168.201.129@tcp [ 8308.252927] Key type .llcrypt unregistered [ 8308.258543] Key type ._llcrypt unregistered