[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 490522420 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002395] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007368] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010133] LSM: Security Framework initializing [ 0.011055] Yama: becoming mindful. [ 0.012046] SELinux: Initializing. [ 0.013070] *** VALIDATE selinux *** [ 0.022181] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027140] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028161] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029125] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030124] *** VALIDATE tmpfs *** [ 0.031471] *** VALIDATE proc *** [ 0.032226] *** VALIDATE cgroup *** [ 0.033011] *** VALIDATE cgroup2 *** [ 0.035047] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036166] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038031] Spectre V2 : User space: Vulnerable [ 0.039010] Speculative Store Bypass: Vulnerable [ 0.042352] debug: unmapping init [mem 0xffffffffb4a59000-0xffffffffb4a60fff] [ 0.044268] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045750] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046032] ... version: 2 [ 0.047015] ... bit width: 48 [ 0.048015] ... generic registers: 4 [ 0.049016] ... value mask: 0000ffffffffffff [ 0.050018] ... max period: 00007fffffffffff [ 0.051018] ... fixed-purpose events: 3 [ 0.052012] ... event mask: 000000070000000f [ 0.053345] rcu: Hierarchical SRCU implementation. [ 0.055721] smp: Bringing up secondary CPUs ... [ 0.056612] x86: Booting SMP configuration: [ 0.057034] .... node #0, CPUs: #1 #2 #3 [ 0.060460] smp: Brought up 1 node, 4 CPUs [ 0.062013] smpboot: Max logical packages: 1 [ 0.063019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.107718] node 0 deferred pages initialised in 41ms [ 0.111267] devtmpfs: initialized [ 0.113275] x86/mm: Memory block size: 128MB [ 0.115994] gcov: version magic: 0x41383552 [ 0.119277] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.122076] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.125347] pinctrl core: initialized pinctrl subsystem [ 0.127178] [ 0.127747] ************************************************************* [ 0.130012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.132012] ** ** [ 0.134010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.137014] ** ** [ 0.139013] ** This means that this kernel is built to expose internal ** [ 0.142015] ** IOMMU data structures, which may compromise security on ** [ 0.144011] ** your system. ** [ 0.147014] ** ** [ 0.149011] ** If you see this message and you are not debugging the ** [ 0.151014] ** kernel, report this immediately to your vendor! ** [ 0.154014] ** ** [ 0.156012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159014] ************************************************************* [ 0.161744] NET: Registered protocol family 16 [ 0.164428] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.167066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.170081] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.174031] cpuidle: using governor menu [ 0.175941] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.178580] PCI: Using configuration type 1 for base access [ 0.182127] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.192117] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.194030] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.198195] cryptd: max_cpu_qlen set to 1000 [ 0.201262] ACPI: Added _OSI(Module Device) [ 0.203021] ACPI: Added _OSI(Processor Device) [ 0.205014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.206025] ACPI: Added _OSI(Processor Aggregator Device) [ 0.212113] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.218498] ACPI: Interpreter enabled [ 0.220066] ACPI: PM: (supports S0 S3 S4 S5) [ 0.221014] ACPI: Using IOAPIC for interrupt routing [ 0.223112] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.227432] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.239106] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.241045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.244025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.248090] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.253383] acpiphp: Slot [2] registered [ 0.256104] acpiphp: Slot [5] registered [ 0.257280] acpiphp: Slot [6] registered [ 0.259160] acpiphp: Slot [7] registered [ 0.261144] acpiphp: Slot [8] registered [ 0.263128] acpiphp: Slot [9] registered [ 0.264162] acpiphp: Slot [10] registered [ 0.265109] acpiphp: Slot [3] registered [ 0.267108] acpiphp: Slot [4] registered [ 0.269152] acpiphp: Slot [11] registered [ 0.270112] acpiphp: Slot [12] registered [ 0.271117] acpiphp: Slot [13] registered [ 0.273119] acpiphp: Slot [14] registered [ 0.275111] acpiphp: Slot [15] registered [ 0.276114] acpiphp: Slot [16] registered [ 0.278150] acpiphp: Slot [17] registered [ 0.280144] acpiphp: Slot [18] registered [ 0.282118] acpiphp: Slot [19] registered [ 0.283116] acpiphp: Slot [20] registered [ 0.285114] acpiphp: Slot [21] registered [ 0.287124] acpiphp: Slot [22] registered [ 0.288147] acpiphp: Slot [23] registered [ 0.290112] acpiphp: Slot [24] registered [ 0.291100] acpiphp: Slot [25] registered [ 0.294234] acpiphp: Slot [26] registered [ 0.295148] acpiphp: Slot [27] registered [ 0.297113] acpiphp: Slot [28] registered [ 0.299111] acpiphp: Slot [29] registered [ 0.300112] acpiphp: Slot [30] registered [ 0.302102] acpiphp: Slot [31] registered [ 0.303048] PCI host bridge to bus 0000:00 [ 0.304019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.306025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.308024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.311026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.314025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.316035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.318324] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.321949] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.325378] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.334787] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.340000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.343021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.345018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.347019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.351463] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.353899] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.357049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.360918] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.367027] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.381024] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.388015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.393333] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.404018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.416018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.432030] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.442916] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.452026] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.459025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.479027] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.493290] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.500020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.514016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.544019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.554993] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.562018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.569020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.587023] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.597152] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.603019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.615015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.634018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.644675] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.651019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.656019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.677019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.689781] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.693364] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.695346] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.699396] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.703267] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.709093] iommu: Default domain type: Passthrough [ 0.710445] SCSI subsystem initialized [ 0.711198] ACPI: bus type USB registered [ 0.712000] usbcore: registered new interface driver usbfs [ 0.713104] usbcore: registered new interface driver hub [ 0.715105] usbcore: registered new device driver usb [ 0.716236] pps_core: LinuxPPS API ver. 1 registered [ 0.718012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.721072] PTP clock support registered [ 0.723055] EDAC MC: Ver: 3.0.0 [ 0.725115] PCI: Using ACPI for IRQ routing [ 0.726893] NetLabel: Initializing [ 0.728011] NetLabel: domain hash size = 128 [ 0.729011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.732091] NetLabel: unlabeled traffic allowed by default [ 0.734291] vgaarb: loaded [ 0.735300] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.737017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.744000] clocksource: Switched to clocksource kvm-clock [ 0.852363] VFS: Disk quotas dquot_6.6.0 [ 0.854067] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.856408] *** VALIDATE ramfs *** [ 0.857603] *** VALIDATE hugetlbfs *** [ 0.859129] pnp: PnP ACPI init [ 0.861627] pnp: PnP ACPI: found 6 devices [ 0.878800] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.882556] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.884653] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.887033] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.889778] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.892057] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.895210] NET: Registered protocol family 2 [ 0.897864] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.902914] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.907058] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.912781] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.916384] TCP: Hash tables configured (established 65536 bind 65536) [ 0.919559] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.922938] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.925976] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.928655] NET: Registered protocol family 1 [ 0.932423] RPC: Registered named UNIX socket transport module. [ 0.934914] RPC: Registered udp transport module. [ 0.936820] RPC: Registered tcp transport module. [ 0.938847] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.941522] NET: Registered protocol family 44 [ 0.944199] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.947127] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.950272] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.953854] PCI: CLS 0 bytes, default 64 [ 0.955921] Unpacking initramfs... [ 2.349896] debug: unmapping init [mem 0xffff8f47bcc54000-0xffff8f47bffbffff] [ 2.354501] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.357129] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.360248] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.867645] Initialise system trusted keyrings [ 2.869934] Key type blacklist registered [ 2.872176] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.883641] zbud: loaded [ 2.887817] *** VALIDATE nfs *** [ 2.889359] *** VALIDATE nfs4 *** [ 2.891266] pstore: using deflate compression [ 2.894497] Platform Keyring initialized [ 3.000914] NET: Registered protocol family 38 [ 3.002680] Key type asymmetric registered [ 3.004047] Asymmetric key parser 'x509' registered [ 3.005772] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.008831] io scheduler mq-deadline registered [ 3.010235] io scheduler kyber registered [ 3.012033] io scheduler bfq registered [ 3.014573] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.017456] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.020793] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.024180] ACPI: Power Button [PWRF] [ 3.029699] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.036612] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.048175] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.054513] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.076736] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.105278] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.134485] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.139711] Non-volatile memory driver v1.3 [ 3.141785] Linux agpgart interface v0.103 [ 3.175287] virtio_blk virtio1: [vda] 146152 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.178330] vda: detected capacity change from 0 to 74829824 [ 3.193510] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.197023] vdb: detected capacity change from 0 to 1073741824 [ 3.212623] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.216253] vdc: detected capacity change from 0 to 2621440000 [ 3.231843] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.235512] vdd: detected capacity change from 0 to 2621440000 [ 3.254510] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.257457] vde: detected capacity change from 0 to 4294967296 [ 3.272583] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.275908] vdf: detected capacity change from 0 to 4294967296 [ 3.282295] libphy: Fixed MDIO Bus: probed [ 3.288544] usbcore: registered new interface driver usbserial_generic [ 3.291524] usbserial: USB Serial support registered for generic [ 3.294212] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.299702] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.301192] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.304234] mousedev: PS/2 mouse device common for all mice [ 3.307581] rtc_cmos 00:05: RTC can wake from S4 [ 3.310324] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.311643] rtc_cmos 00:05: registered as rtc0 [ 3.317243] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.317902] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.322329] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.323096] intel_pstate: CPU model not supported [ 3.328264] hid: raw HID events driver (C) Jiri Kosina [ 3.330813] usbcore: registered new interface driver usbhid [ 3.333076] usbhid: USB HID core driver [ 3.335249] drop_monitor: Initializing network drop monitor service [ 3.338088] Initializing XFRM netlink socket [ 3.340116] NET: Registered protocol family 10 [ 3.343676] Segment Routing with IPv6 [ 3.345403] NET: Registered protocol family 17 [ 3.347819] mpls_gso: MPLS GSO support [ 3.352527] RAS: Correctable Errors collector initialized. [ 3.354906] AVX version of gcm_enc/dec engaged. [ 3.356447] AES CTR mode by8 optimization enabled [ 3.436289] sched_clock: Marking stable (3436243522, 0)->(4415067034, -978823512) [ 3.439945] registered taskstats version 1 [ 3.442381] Loading compiled-in X.509 certificates [ 3.445353] zswap: loaded using pool lzo/zbud [ 3.470493] Key type big_key registered [ 3.484659] Key type encrypted registered [ 3.486206] ima: No TPM chip found, activating TPM-bypass! [ 3.487962] ima: Allocated hash algorithm: sha1 [ 3.489707] ima: No architecture policies found [ 3.491172] evm: Initialising EVM extended attributes: [ 3.492806] evm: security.selinux [ 3.493929] evm: security.ima [ 3.494862] evm: security.capability [ 3.495921] evm: HMAC attrs: 0x1 [ 3.498174] rtc_cmos 00:05: setting system clock to 2026-08-27 08:06:03 UTC (1787817963) [ 3.504100] debug: unmapping init [mem 0xffffffffb5a03000-0xffffffffb5bfffff] [ 3.506813] debug: unmapping init [mem 0xffffffffb4782000-0xffffffffb4a58fff] [ 3.515095] Write protecting the kernel read-only data: 28672k [ 3.518156] debug: unmapping init [mem 0xffffffffb2e03000-0xffffffffb2ffffff] [ 3.520742] debug: unmapping init [mem 0xffffffffb3714000-0xffffffffb37fffff] [ 3.552196] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.561410] systemd[1]: Detected virtualization kvm. [ 3.563448] systemd[1]: Detected architecture x86-64. [ 3.565555] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.596656] systemd[1]: No hostname configured. [ 3.598292] systemd[1]: Set hostname to . [ 3.600595] random: systemd: uninitialized urandom read (16 bytes read) [ 3.603393] systemd[1]: Initializing machine ID from random generator. [ 3.748816] random: systemd: uninitialized urandom read (16 bytes read) [ 3.752331] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.758669] random: systemd: uninitialized urandom read (16 bytes read) [ 3.762198] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.768333] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Local Encrypted Volumes. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.475736] device-mapper: uevent: version 1.0.3 [ 4.478516] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.290486] virtio_net virtio0 ens2: renamed from eth0 [ 5.309532] random: fast init done [ 5.483796] scsi host0: ata_piix [ 5.497441] scsi host1: ata_piix [ 5.499243] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.502265] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.894324] dracut-initqueue[585]: RTNETLINK answers: File exists [ 10.001602] random: crng init done [ 10.003051] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.635833] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.883367] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.155799] SELinux: Disabled at runtime. [ 12.222327] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.231865] systemd[1]: Detected virtualization kvm. [ 12.233784] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.742616] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.747382] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.753195] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.758343] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.762362] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.771928] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.780133] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ 12.908484] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Slices. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.301990] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.692711] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.769373] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.899360] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.911958] EDAC sbridge: Ver: 1.1.2 [ 15.601502] Key type dns_resolver registered [ 15.921411] NFS: Registering the id_resolver key type [ 15.923576] Key type id_resolver registered [ 15.925283] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg404-server login: [ 90.810013] hrtimer: interrupt took 2611992 ns [ 90.943183] libcfs: loading out-of-tree module taints kernel. [ 91.031387] Key type ._llcrypt registered [ 91.035222] Key type .llcrypt registered [ 91.217263] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_hostid [ 110.858813] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 113.045603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 113.080663] alg: No test for adler32 (adler32-zlib) [ 114.471263] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 115.457293] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 117.383436] Key type lgssc registered [ 120.081790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 144.412818] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 193.112853] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 214.276973] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 214.328158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 215.655773] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 215.687911] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 215.755209] Lustre: lustre-MDT0000: new disk, initializing [ 215.816611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 215.837313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 221.445243] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 237.384072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 237.693780] Lustre: 6513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 237.775764] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 237.798904] Lustre: Skipped 1 previous similar message [ 237.995599] Lustre: lustre-MDT0001: new disk, initializing [ 238.105650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 238.165397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 238.178220] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 243.345654] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 248.277549] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 257.983584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 258.321249] Lustre: lustre-OST0000: new disk, initializing [ 258.329660] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 258.338440] Lustre: 8453:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 258.422914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 264.746515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 264.763180] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 264.920752] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 265.744159] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 283.009286] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 283.106147] Lustre: lustre-OST0001: new disk, initializing [ 283.109720] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 283.115800] Lustre: 9525:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 283.177757] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 291.074255] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 291.387475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 291.400356] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 291.667776] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 307.870900] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 315.858969] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 325.269276] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing check_logdir /tmp/testlogs/ [ 332.383104] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing yml_node [ 336.933638] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 339.514883] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 342.540503] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 344.951037] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 04:11:41 EDT 2026 [ 371.310309] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 373.130337] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 391.649137] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 391.666923] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 391.700903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 394.209935] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 394.213893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 394.235090] Lustre: Skipped 2 previous similar messages [ 394.257438] Lustre: Skipped 3 previous similar messages [ 396.060544] Lustre: server umount lustre-MDT0000 complete [ 400.837157] LustreError: 6507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787818360 with bad export cookie 13476283350023795579 [ 400.841511] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 400.868874] LustreError: 6507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 404.449546] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 404.452941] LustreError: 10101:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 404.453549] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 404.473068] Lustre: Skipped 1 previous similar message [ 404.503158] LustreError: 10101:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 405.473250] LustreError: 6521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 405.483227] LustreError: 6521:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 410.594234] LustreError: 6523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 410.594710] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 410.650561] Lustre: Skipped 4 previous similar messages [ 415.200465] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 415.630099] Lustre: server umount lustre-MDT0001 complete [ 420.864283] Lustre: server umount lustre-OST0000 complete [ 425.613616] Lustre: server umount lustre-OST0001 complete [ 434.405924] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_hostid [ 447.621537] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 489.446320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 489.749846] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 489.792887] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 489.891261] Lustre: lustre-MDT0000: new disk, initializing [ 489.981575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 489.999606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 494.754728] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 506.488633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 506.689920] Lustre: 15101:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 506.780259] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 506.797569] Lustre: Skipped 1 previous similar message [ 506.899833] Lustre: lustre-MDT0001: new disk, initializing [ 507.044539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 507.077144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 507.096593] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 511.633941] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 521.121366] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 526.618651] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 534.402476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 534.824647] Lustre: lustre-OST0000: new disk, initializing [ 534.829423] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 534.837609] Lustre: 17066:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 534.947751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 536.728275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 536.739062] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 536.917257] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 541.762379] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 552.386981] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 557.289130] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 557.551428] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 561.823243] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 562.046588] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 567.784425] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 567.784879] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 567.793879] LustreError: Skipped 1 previous similar message [ 567.802671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 567.803074] Lustre: Skipped 1 previous similar message [ 567.822057] Lustre: Skipped 1 previous similar message [ 570.103095] Lustre: server umount lustre-OST0000 complete [ 578.018136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 578.034218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 578.039660] Lustre: Skipped 1 previous similar message [ 578.059952] Lustre: Skipped 1 previous similar message [ 588.255256] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 588.440700] Lustre: server umount lustre-MDT0000 complete [ 592.235840] LustreError: 15092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787818552 with bad export cookie 13476283350023797112 [ 592.242856] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 592.260574] LustreError: 15092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 592.547026] Lustre: server umount lustre-MDT0001 complete [ 595.731956] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 04:15:53 (1787818553) [ 602.587858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 603.162959] LustreError: 19333:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 603.180483] LustreError: 19333:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 603.298884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 608.270136] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 608.739028] LustreError: 19334:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 617.977654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 623.429614] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 631.899851] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 637.514785] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 645.539528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 654.443318] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 664.130602] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 673.772930] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 673.775041] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 673.792046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 673.796228] LustreError: Skipped 1 previous similar message [ 673.812298] Lustre: Skipped 5 previous similar messages [ 678.683583] Lustre: server umount lustre-OST0000 complete [ 688.997800] Lustre: server umount lustre-MDT0000 complete [ 689.121633] LustreError: 19330:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 689.149385] LustreError: 19330:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 692.392376] LustreError: 19315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787818652 with bad export cookie 13476283350023797987 [ 692.398653] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 692.407788] LustreError: 19315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 692.723648] Lustre: server umount lustre-MDT0001 complete [ 704.496976] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 707.659800] Key type lgssc unregistered [ 707.901269] LNet: 22871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 707.916329] LNetError: 22871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 707.937765] LNet: Removed LNI 192.168.204.104@tcp [ 709.116184] Key type .llcrypt unregistered [ 709.118816] Key type ._llcrypt unregistered [ 736.135509] Key type ._llcrypt registered [ 736.143952] Key type .llcrypt registered [ 736.256769] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 737.998263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 738.074112] alg: No test for adler32 (adler32-zlib) [ 739.373669] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 739.830639] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 741.519176] Key type lgssc registered [ 743.496432] Lustre: Echo OBD driver; http://www.lustre.org/ [ 753.693121] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 753.763376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 755.803537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 762.114088] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 765.924345] LustreError: 24285:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 765.964235] LustreError: 24285:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 771.044152] LustreError: 24286:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 772.482384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 772.982296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 778.595209] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 787.859336] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 794.119748] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 801.065569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 801.565930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 807.607097] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 814.832390] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 825.149893] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 835.553261] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 835.557204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 835.571827] Lustre: Skipped 1 previous similar message [ 835.589504] Lustre: Skipped 1 previous similar message [ 840.676666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 840.688585] Lustre: Skipped 1 previous similar message [ 841.077741] Lustre: server umount lustre-OST0000 complete [ 844.769991] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 844.775776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 844.786058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 845.794625] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 850.914729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 850.927155] Lustre: Skipped 3 previous similar messages [ 856.034434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 856.045726] Lustre: Skipped 1 previous similar message [ 859.103419] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 859.309773] Lustre: server umount lustre-MDT0000 complete [ 861.152969] LustreError: 24286:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 861.177765] LustreError: 24286:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 863.456980] LustreError: 24266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787818823 with bad export cookie 7569786002256440257 [ 863.466168] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 863.469232] LustreError: 24266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 863.885338] Lustre: server umount lustre-MDT0001 complete [ 876.997632] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 880.389228] Key type lgssc unregistered [ 880.772802] LNet: 28268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 880.794079] LNetError: 28268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 880.809852] LNet: Removed LNI 192.168.204.104@tcp [ 881.800643] Key type .llcrypt unregistered [ 881.804930] Key type ._llcrypt unregistered [ 905.518399] Key type ._llcrypt registered [ 905.525907] Key type .llcrypt registered [ 905.634712] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 907.012570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 907.051409] alg: No test for adler32 (adler32-zlib) [ 908.203142] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 908.539730] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 910.234154] Key type lgssc registered [ 911.416982] Lustre: Echo OBD driver; http://www.lustre.org/ [ 920.972492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 921.001829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 922.663181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 929.067938] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 932.845659] LustreError: 29687:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 932.878120] LustreError: 29687:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 937.955430] LustreError: 29686:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 940.224825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 940.833512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 946.363091] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 955.798191] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 963.167103] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 977.061386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 977.454971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 983.855223] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 992.834679] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 992.940567] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1000.928185] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1000.940755] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1000.965229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1002.978289] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1002.980756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1003.011292] Lustre: Skipped 1 previous similar message [ 1007.187899] Lustre: server umount lustre-OST0000 complete [ 1012.709229] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1012.723851] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1012.755441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1017.140141] Lustre: server umount lustre-MDT0000 complete [ 1018.337707] LustreError: 29687:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1018.352730] LustreError: 29687:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1021.577770] LustreError: 29668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787818981 with bad export cookie 4551008003770888307 [ 1021.591667] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1021.593207] LustreError: 29668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1022.154965] Lustre: server umount lustre-MDT0001 complete [ 1034.648975] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 1037.893323] Key type lgssc unregistered [ 1038.180798] LNet: 33570:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1038.189195] LNetError: 33570:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1039.207665] LNet: Removed LNI 192.168.204.104@tcp [ 1040.258760] Key type .llcrypt unregistered [ 1040.261177] Key type ._llcrypt unregistered [ 1067.623849] Key type ._llcrypt registered [ 1067.627639] Key type .llcrypt registered [ 1067.781737] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 1069.500755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1069.538703] alg: No test for adler32 (adler32-zlib) [ 1070.582583] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 1070.883876] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 1072.593108] Key type lgssc registered [ 1073.951550] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1082.295313] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1082.343347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1083.930956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1088.734862] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1093.110323] LustreError: 34984:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1093.144239] LustreError: 34984:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1098.226651] LustreError: 34985:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1098.732716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1099.280403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1104.196524] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1112.330360] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1119.710380] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1127.212032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1127.568122] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1128.638253] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1134.989437] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1144.311137] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1154.529546] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1154.541690] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1154.569273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1155.555136] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1155.556235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1155.584394] Lustre: Skipped 1 previous similar message [ 1160.133421] Lustre: server umount lustre-OST0000 complete [ 1165.803285] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1165.805666] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1165.825717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1165.841595] Lustre: Skipped 1 previous similar message [ 1165.860427] Lustre: Skipped 1 previous similar message [ 1170.413316] Lustre: server umount lustre-MDT0000 complete [ 1170.912260] LustreError: 34985:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1170.946813] LustreError: 34985:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1175.013775] LustreError: 34984:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1175.060391] LustreError: 34984:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1175.255906] LustreError: 34967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787819135 with bad export cookie 7007362681604487312 [ 1175.262049] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1175.274492] LustreError: 34967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1175.758743] Lustre: server umount lustre-MDT0001 complete [ 1188.376510] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 1192.474453] Key type lgssc unregistered [ 1192.918606] LNet: 38525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1192.938135] LNetError: 38525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1192.958867] LNet: Removed LNI 192.168.204.104@tcp [ 1194.196102] Key type .llcrypt unregistered [ 1194.198950] Key type ._llcrypt unregistered [ 1222.558694] Key type ._llcrypt registered [ 1222.560276] Key type .llcrypt registered [ 1222.662552] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 1223.544094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1223.555705] alg: No test for adler32 (adler32-zlib) [ 1224.698247] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 1224.898757] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 1226.599284] Key type lgssc registered [ 1227.706690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1235.611237] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1235.640163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1237.087689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1241.615777] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1247.715080] LustreError: 39942:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1247.750178] LustreError: 39942:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1250.975718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1251.319955] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1255.912918] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1264.145048] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1271.226599] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1278.523863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1278.894633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1283.030300] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1288.967275] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1299.119117] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1306.606738] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1306.613696] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1306.614893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1306.633354] Lustre: Skipped 1 previous similar message [ 1307.375747] Lustre: lustre-OST0000: Not available for connect from 192.168.204.4@tcp (stopping) [ 1307.388162] Lustre: Skipped 1 previous similar message [ 1311.716108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1311.729951] Lustre: Skipped 1 previous similar message [ 1312.461179] LustreError: 41763:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1312.599981] Lustre: server umount lustre-OST0000 complete [ 1336.806836] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1336.830313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1336.840665] Lustre: Skipped 1 previous similar message [ 1341.921134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1341.930486] Lustre: Skipped 1 previous similar message [ 1349.087347] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1349.347256] Lustre: server umount lustre-MDT0000 complete [ 1352.160772] LustreError: 41671:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1352.190198] LustreError: 41671:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1353.480472] LustreError: 39921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787819313 with bad export cookie 10444210192387261207 [ 1353.487474] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1353.490659] LustreError: 39921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1360.024635] Lustre: server umount lustre-MDT0001 complete [ 1373.174637] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 1376.081788] Key type lgssc unregistered [ 1376.372273] LNet: 43657:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1376.384912] LNetError: 43657:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1376.402913] LNet: Removed LNI 192.168.204.104@tcp [ 1377.234241] Key type .llcrypt unregistered [ 1377.237468] Key type ._llcrypt unregistered [ 1404.200220] Key type ._llcrypt registered [ 1404.205032] Key type .llcrypt registered [ 1404.339563] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 1405.804123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1405.882073] alg: No test for adler32 (adler32-zlib) [ 1407.036558] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 1407.331707] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 1409.040491] Key type lgssc registered [ 1410.164794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1418.686800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1418.708664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1420.142865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1425.146666] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1430.499871] LustreError: 45072:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1430.527634] LustreError: 45072:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1435.156429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1435.502154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1440.061276] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1450.111498] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1456.849476] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1464.976037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1465.344380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1470.465848] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1472.163066] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1481.523845] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1488.626819] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.4@tcp (stopping) [ 1490.918404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1490.926488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1490.933710] Lustre: Skipped 2 previous similar messages [ 1490.958075] Lustre: Skipped 2 previous similar messages [ 1493.727313] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.4@tcp (stopping) [ 1494.543985] Lustre: server umount lustre-MDT0000 complete [ 1496.041223] LustreError: 45800:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1496.058858] LustreError: 45800:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1498.834932] LustreError: 45069:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1500.080497] LustreError: 45053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787819460 with bad export cookie 14450030409715958528 [ 1500.088080] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1500.095411] LustreError: 45053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1501.154078] LustreError: 45067:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1501.159787] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1501.201650] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1505.185504] LustreError: 45068:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1505.201777] LustreError: 45068:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1506.635770] Lustre: server umount lustre-MDT0001 complete [ 1528.354422] Lustre: server umount lustre-OST0000 complete [ 1547.077207] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 1550.266621] Key type lgssc unregistered [ 1550.552681] LNet: 48935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1550.575248] LNetError: 48935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1550.606621] LNet: Removed LNI 192.168.204.104@tcp [ 1551.508191] Key type .llcrypt unregistered [ 1551.513652] Key type ._llcrypt unregistered [ 1578.653909] Key type ._llcrypt registered [ 1578.657870] Key type .llcrypt registered [ 1578.786627] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 1579.947093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1579.977383] alg: No test for adler32 (adler32-zlib) [ 1581.131817] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 1581.373607] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 1583.039277] Key type lgssc registered [ 1584.355357] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1592.374247] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1592.396265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1607.903696] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1613.023318] LustreError: 50322:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.104@tcp: failed processing log, type 4: rc = -110 [ 1639.712931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1645.895992] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1654.619123] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1820.383107] Lustre: server umount lustre-OST0000 complete [ 1838.501501] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 1841.500325] Key type lgssc unregistered [ 1841.907472] LNet: 52001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1841.920306] LNetError: 52001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1841.940366] LNet: Removed LNI 192.168.204.104@tcp [ 1842.850349] Key type .llcrypt unregistered [ 1842.854722] Key type ._llcrypt unregistered [ 1868.025550] Key type ._llcrypt registered [ 1868.028299] Key type .llcrypt registered [ 1868.158945] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 1869.195744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1869.215678] alg: No test for adler32 (adler32-zlib) [ 1870.438431] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 1870.733904] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 1872.463171] Key type lgssc registered [ 1873.722393] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1881.436632] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1881.470158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1883.019340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1887.957518] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1892.333976] LustreError: 53415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1892.367330] LustreError: 53415:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1896.853287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1897.233045] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1901.775308] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1910.028625] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1916.129754] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1924.222678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1924.554923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1927.620101] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1931.752086] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 1939.928904] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1943.024751] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1943.028861] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1943.060167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1944.106589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1949.158860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1949.180272] Lustre: Skipped 2 previous similar messages [ 1954.282693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1954.296739] Lustre: Skipped 1 previous similar message [ 1957.344384] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1957.543636] Lustre: server umount lustre-OST0000 complete [ 1964.001455] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1964.015583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1964.032939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1967.988691] Lustre: server umount lustre-MDT0000 complete [ 1969.632033] LustreError: 53415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1969.659409] LustreError: 53415:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1972.220298] LustreError: 53397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787819932 with bad export cookie 6450816428697386449 [ 1972.227261] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1972.235371] LustreError: 53397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1972.822910] Lustre: server umount lustre-MDT0001 complete [ 1985.835538] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 1989.014589] Key type lgssc unregistered [ 1989.427844] LNet: 56953:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1989.440023] LNetError: 56953:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1989.458845] LNet: Removed LNI 192.168.204.104@tcp [ 1990.498675] Key type .llcrypt unregistered [ 1990.503041] Key type ._llcrypt unregistered [ 2016.643711] Key type ._llcrypt registered [ 2016.645790] Key type .llcrypt registered [ 2016.757254] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 2017.976733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2018.139384] alg: No test for adler32 (adler32-zlib) [ 2019.266579] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 2019.728121] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 2021.519919] Key type lgssc registered [ 2022.993215] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2031.517894] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2031.544953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2047.135574] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2052.255350] LustreError: 58341:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.104@tcp: failed processing log, type 4: rc = -110 [ 2078.943884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2084.963745] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 2092.589359] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2099.669827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2100.429827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2100.524289] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2105.666360] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 2105.861245] LustreError: 58366:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2105.903767] LustreError: 58366:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2110.945415] LustreError: 58367:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2113.019374] LustreError: 59378:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2113.049539] LustreError: 59378:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2114.891796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2115.424139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2121.847740] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 2131.730492] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2139.228354] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2141.665753] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2141.678941] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2141.704873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2146.282517] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2146.283369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2146.288730] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2146.317956] Lustre: Skipped 1 previous similar message [ 2147.665739] Lustre: server umount lustre-OST0000 complete [ 2171.872353] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2171.873262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2171.894539] Lustre: Skipped 1 previous similar message [ 2171.918882] Lustre: Skipped 1 previous similar message [ 2176.327836] Lustre: server umount lustre-MDT0000 complete [ 2176.993491] LustreError: 59378:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2177.035160] LustreError: 59378:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2181.046922] LustreError: 58347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787820141 with bad export cookie 18152178374924195859 [ 2181.057798] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2181.064135] LustreError: 58347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2187.426630] Lustre: server umount lustre-MDT0001 complete [ 2199.480631] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 2202.536277] Key type lgssc unregistered [ 2202.866764] LNet: 62088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2202.878593] LNetError: 62088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2202.898622] LNet: Removed LNI 192.168.204.104@tcp [ 2203.905651] Key type .llcrypt unregistered [ 2203.916913] Key type ._llcrypt unregistered [ 2228.348345] Key type ._llcrypt registered [ 2228.350925] Key type .llcrypt registered [ 2228.462647] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 2229.807271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2229.874336] alg: No test for adler32 (adler32-zlib) [ 2231.007099] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 2231.258583] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 2232.944103] Key type lgssc registered [ 2234.117964] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2242.661659] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2242.686280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2244.354525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2249.108937] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 2254.818241] LustreError: 63500:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2254.839672] LustreError: 63500:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2259.258813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2259.827391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2265.127399] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 2273.107078] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2278.441433] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2286.055668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2286.488872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2289.610934] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2293.346755] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 2302.278664] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2323.437225] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2323.451100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2323.467829] Lustre: Skipped 1 previous similar message [ 2323.497216] Lustre: Skipped 1 previous similar message [ 2327.822587] Lustre: server umount lustre-OST0000 complete [ 2333.668055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2333.690311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2333.698339] Lustre: Skipped 1 previous similar message [ 2338.428982] Lustre: server umount lustre-MDT0000 complete [ 2338.786299] LustreError: 65753:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2338.813168] LustreError: 65753:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2342.360338] LustreError: 63478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787820302 with bad export cookie 16671123128239487345 [ 2342.366485] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2342.376862] LustreError: 63478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2342.701649] Lustre: server umount lustre-MDT0001 complete [ 2354.124412] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 2356.699991] Key type lgssc unregistered [ 2356.995525] LNet: 67061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2357.005869] LNetError: 67061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2357.033716] LNet: Removed LNI 192.168.204.104@tcp [ 2357.903660] Key type .llcrypt unregistered [ 2357.907323] Key type ._llcrypt unregistered [ 2384.013960] Key type ._llcrypt registered [ 2384.016478] Key type .llcrypt registered [ 2384.136260] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 2385.080414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2385.102945] alg: No test for adler32 (adler32-zlib) [ 2386.150660] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 2386.464887] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 2388.168865] Key type lgssc registered [ 2389.466825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2399.147579] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 04:45:56 (1787820356) [ 2406.450871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2406.510278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2408.211306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2412.900550] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 2418.673253] LustreError: 68765:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2418.702142] LustreError: 68765:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2421.887791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2422.454491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2427.465685] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 2435.505603] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2442.456496] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2449.960041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2450.424660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2453.514760] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2456.752549] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 2464.662877] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2473.616314] Lustre: Failing over lustre-MDT0000 [ 2473.884406] Lustre: server umount lustre-MDT0000 complete [ 2473.953913] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2473.956038] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2473.966223] LustreError: 68762:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2473.975036] Lustre: Skipped 1 previous similar message [ 2474.002594] LustreError: 68762:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2476.776521] LustreError: 68761:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2479.075239] LustreError: 68760:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2479.100797] LustreError: 68760:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2484.197633] LustreError: 69494:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2484.217678] LustreError: 69494:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2484.465131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2484.618797] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2484.835332] Lustre: *** cfs_fail_loc=135, val=0*** [ 2484.979112] LustreError: 71852:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2484.990914] LustreError: 71852:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.104@tcp: cfg command failed: rc = -2 [ 2485.004783] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2485.004783] [ 2485.020543] LustreError: MGC192.168.204.104@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2485.045354] LustreError: 71841:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2485.052954] LustreError: 71841:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 2485.061366] LustreError: 71841:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2485.069133] LustreError: 71841:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2485.091771] Lustre: server umount lustre-MDT0000 complete [ 2485.095483] LustreError: 71841:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2492.756493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2492.925404] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2493.339791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2493.423374] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2497.222083] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2497.552445] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 2498.545080] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2498.628557] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2498.708991] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2508.773584] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2508.787683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2508.801896] Lustre: Skipped 2 previous similar messages [ 2508.815994] Lustre: Skipped 1 previous similar message [ 2511.838041] Lustre: server umount lustre-OST0000 complete [ 2519.009674] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2519.011136] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2519.012446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2519.056344] Lustre: Skipped 1 previous similar message [ 2524.129178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2524.135413] Lustre: Skipped 2 previous similar messages [ 2528.226234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2530.271120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2530.541835] Lustre: server umount lustre-MDT0000 complete [ 2533.357727] LustreError: 68765:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2533.383360] LustreError: 68765:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 2535.833445] LustreError: 68748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787820495 with bad export cookie 246084416641957345 [ 2535.840403] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2535.856056] LustreError: 68748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2536.326393] Lustre: server umount lustre-MDT0001 complete [ 2549.118623] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 2552.287970] Key type lgssc unregistered [ 2552.591836] LNet: 73634:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2552.601039] LNetError: 73634:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2553.639947] LNet: Removed LNI 192.168.204.104@tcp [ 2554.807610] Key type .llcrypt unregistered [ 2554.810082] Key type ._llcrypt unregistered [ 2581.350133] Key type ._llcrypt registered [ 2581.353099] Key type .llcrypt registered [ 2581.460866] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 2582.194784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2582.246148] alg: No test for adler32 (adler32-zlib) [ 2583.452899] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 2583.741204] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 2585.415199] Key type lgssc registered [ 2586.482517] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2595.051855] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2595.062330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2596.671383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2601.627943] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 2606.050746] LustreError: 75061:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2606.081548] LustreError: 75061:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2611.180238] LustreError: 75062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2611.680115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2612.203468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2616.952988] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 2626.145326] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2633.290242] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2641.369216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2641.728046] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2642.767570] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2649.605737] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 2660.102706] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2667.604493] Lustre: Failing over lustre-MDT0000 [ 2668.074775] Lustre: server umount lustre-MDT0000 complete [ 2668.284159] LustreError: 76705:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2668.516513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2668.530646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2672.610886] LustreError: 75062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2672.614310] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2672.628914] LustreError: 75062:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2677.741255] LustreError: 75062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2677.789363] LustreError: 75062:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2680.831753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2680.994366] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2681.297960] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2681.496931] LustreError: 78151:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2681.515164] LustreError: 78151:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.104@tcp: cfg command failed: rc = -9 [ 2681.540067] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2681.540067] [ 2681.567392] LustreError: MGC192.168.204.104@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2681.592292] LustreError: 78139:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2681.606513] LustreError: 78139:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -9 [ 2681.623182] LustreError: 78139:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2681.630979] LustreError: 78139:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2681.672702] Lustre: server umount lustre-MDT0000 complete [ 2681.675218] LustreError: 78139:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2687.971985] LustreError: 75062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2688.005336] LustreError: 75062:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 2690.374244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2690.583165] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2690.601291] LustreError: 75053:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2691.077901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2691.190501] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2693.861642] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2696.207744] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2696.256933] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2696.332226] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2698.297819] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 2711.527588] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2711.528867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2711.537043] Lustre: Skipped 2 previous similar messages [ 2715.722662] Lustre: server umount lustre-OST0000 complete [ 2721.760130] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2721.773697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2721.780350] Lustre: Skipped 1 previous similar message [ 2721.794790] Lustre: Skipped 2 previous similar messages [ 2726.880473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2726.894717] Lustre: Skipped 1 previous similar message [ 2727.311180] Lustre: server umount lustre-MDT0000 complete [ 2731.875496] LustreError: 75042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787820691 with bad export cookie 12372366070490512406 [ 2731.877216] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2731.890857] LustreError: 75042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2732.165637] Lustre: server umount lustre-MDT0001 complete [ 2746.400566] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 2750.002862] Key type lgssc unregistered [ 2750.318633] LNet: 79936:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2750.328033] LNetError: 79936:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2751.398271] LNet: Removed LNI 192.168.204.104@tcp [ 2752.590415] Key type .llcrypt unregistered [ 2752.592851] Key type ._llcrypt unregistered [ 2767.036363] Key type ._llcrypt registered [ 2767.039896] Key type .llcrypt registered [ 2767.144570] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 2767.453350] Key type .llcrypt unregistered [ 2767.456414] Key type ._llcrypt unregistered [ 2793.921582] Key type ._llcrypt registered [ 2793.931756] Key type .llcrypt registered [ 2794.190771] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 2796.184443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2796.272460] alg: No test for adler32 (adler32-zlib) [ 2797.544629] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 2797.899731] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 2799.583185] Key type lgssc registered [ 2801.185372] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2811.545569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2811.577488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2813.118812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2818.354138] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 2823.683609] LustreError: 81841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2823.721835] LustreError: 81841:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2828.775362] LustreError: 81842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2829.040151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2829.509252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2835.208702] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 2844.780065] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2851.864952] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2860.230340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2860.494849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2868.387851] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 2876.903543] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2884.026360] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2897.389418] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2897.397120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2897.417373] Lustre: Skipped 1 previous similar message [ 2897.436379] Lustre: Skipped 1 previous similar message [ 2902.503645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2902.512159] Lustre: Skipped 1 previous similar message [ 2902.786796] Lustre: server umount lustre-OST0000 complete [ 2907.634858] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2907.656770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2907.668053] Lustre: Skipped 1 previous similar message [ 2912.737886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2912.750398] Lustre: Skipped 1 previous similar message [ 2913.403661] Lustre: server umount lustre-MDT0000 complete [ 2917.864214] LustreError: 81842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2917.890704] LustreError: 81842:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2917.938117] LustreError: 81823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787820877 with bad export cookie 3600835169723570280 [ 2917.941846] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2917.954848] LustreError: 81823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2918.411380] Lustre: server umount lustre-MDT0001 complete [ 2933.382147] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 2937.612816] Key type lgssc unregistered [ 2937.984799] LNet: 85404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2937.998323] LNetError: 85404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2938.024213] LNet: Removed LNI 192.168.204.104@tcp [ 2939.387362] Key type .llcrypt unregistered [ 2939.389136] Key type ._llcrypt unregistered [ 2969.011242] Key type ._llcrypt registered [ 2969.013244] Key type .llcrypt registered [ 2969.121288] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 2970.344221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2970.403397] alg: No test for adler32 (adler32-zlib) [ 2971.617083] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 2971.886831] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 2973.647219] Key type lgssc registered [ 2975.258703] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2985.950696] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2985.988704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2987.682676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2993.943916] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 2997.216657] LustreError: 86820:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2997.241893] LustreError: 86820:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3002.341185] LustreError: 86821:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3004.653321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3005.172518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3010.491621] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 3020.356901] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3026.993467] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3035.083950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3035.351765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3041.898627] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 3051.178646] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3057.917865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 3071.427117] Lustre: server umount lustre-OST0000 complete [ 3077.095781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3077.111588] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3077.139369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3078.625752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3078.627868] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3083.751327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3083.765390] Lustre: Skipped 2 previous similar messages [ 3087.647597] Lustre: 86130:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787821031/real 1787821031] req@000000004a1f95ed x1874666120575360/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787821047 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3087.684559] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3087.705520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3087.714754] Lustre: Skipped 1 previous similar message [ 3089.887236] Lustre: 86129:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787821033/real 1787821033] req@00000000fb1a6463 x1874666120576384/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787821049 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3089.889102] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3090.179566] Lustre: server umount lustre-MDT0000 complete [ 3092.960497] LustreError: 86814:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3092.998940] LustreError: 86814:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3094.047141] Lustre: 86129:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787821038/real 1787821038] req@00000000019c6964 x1874666120578176/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787821054 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3095.299570] LustreError: 86801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787821055 with bad export cookie 16426709218707849960 [ 3095.312505] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3095.317022] LustreError: 86801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3095.663963] Lustre: server umount lustre-MDT0001 complete [ 3108.520485] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 3111.497932] Key type lgssc unregistered [ 3111.911652] LNet: 90382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3111.919815] LNetError: 90382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3111.932418] LNet: Removed LNI 192.168.204.104@tcp [ 3112.839488] Key type .llcrypt unregistered [ 3112.843075] Key type ._llcrypt unregistered [ 3139.624643] Key type ._llcrypt registered [ 3139.629931] Key type .llcrypt registered [ 3139.786941] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 3141.448891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3141.495289] alg: No test for adler32 (adler32-zlib) [ 3142.648768] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 3142.964817] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 3144.687639] Key type lgssc registered [ 3146.376149] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3155.859324] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3155.892499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3157.515818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3163.201472] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 3167.718083] LustreError: 91797:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3167.739694] LustreError: 91797:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3172.835642] LustreError: 91798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3173.758267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3174.180484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3179.440830] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 3188.977500] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3196.037529] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3204.799215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3205.227484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3213.778198] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 3223.334889] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3227.844257] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3241.964594] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3241.978444] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3241.997746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3243.491477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3243.494102] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3247.673868] Lustre: server umount lustre-OST0000 complete [ 3253.735927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3253.742506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3253.749346] Lustre: Skipped 1 previous similar message [ 3253.772121] Lustre: Skipped 2 previous similar messages [ 3258.308036] Lustre: server umount lustre-MDT0000 complete [ 3258.850671] LustreError: 91798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3258.885474] LustreError: 91798:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3263.182223] LustreError: 91778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787821223 with bad export cookie 3140261199361528974 [ 3263.188535] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3263.195891] LustreError: 91778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3263.661117] Lustre: server umount lustre-MDT0001 complete [ 3277.278086] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 3280.625319] Key type lgssc unregistered [ 3281.034105] LNet: 95357:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3281.047064] LNetError: 95357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3281.064892] LNet: Removed LNI 192.168.204.104@tcp [ 3282.073424] Key type .llcrypt unregistered [ 3282.076301] Key type ._llcrypt unregistered [ 3309.655344] Key type ._llcrypt registered [ 3309.658945] Key type .llcrypt registered [ 3309.762483] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 3311.102857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3311.133629] alg: No test for adler32 (adler32-zlib) [ 3312.396806] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 3312.713184] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 3314.431249] Key type lgssc registered [ 3315.662693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3324.746905] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3324.782202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3340.383435] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3345.503452] LustreError: 96758:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.104@tcp: failed processing log, type 4: rc = -110 [ 3372.191220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3380.811219] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 3390.381364] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3396.469818] Lustre: server umount lustre-OST0000 complete [ 3406.031980] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 05:02:43 (1787821363) [ 3413.514365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3414.184176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3419.633499] LustreError: 98397:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3419.669351] LustreError: 98397:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3420.853610] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 3424.741270] LustreError: 98398:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3429.857182] LustreError: 98397:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3432.736632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3433.651233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3438.983857] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 3447.947190] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3454.445309] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3462.229882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3462.716037] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3469.271160] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 3478.092217] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3485.234843] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3503.591323] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3503.602679] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3503.616117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3505.634635] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3505.635780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3505.677629] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3508.031563] Lustre: server umount lustre-OST0000 complete [ 3515.363276] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3515.376622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3515.395729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3515.400540] Lustre: Skipped 1 previous similar message [ 3518.990893] Lustre: server umount lustre-MDT0000 complete [ 3520.992238] LustreError: 99832:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3521.020223] LustreError: 99832:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3523.147723] LustreError: 98378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787821483 with bad export cookie 11436010767740330598 [ 3523.159458] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3523.593796] Lustre: server umount lustre-MDT0001 complete [ 3535.564403] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 3539.225358] Key type lgssc unregistered [ 3539.610890] LNet: 102119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3539.640293] LNetError: 102119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3540.736023] LNet: Removed LNI 192.168.204.104@tcp [ 3541.970526] Key type .llcrypt unregistered [ 3541.972863] Key type ._llcrypt unregistered [ 3589.698271] Key type ._llcrypt registered [ 3589.705318] Key type .llcrypt registered [ 3589.855721] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 3591.486954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3591.719139] alg: No test for adler32 (adler32-zlib) [ 3593.023425] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 3593.469619] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 3595.271168] Key type lgssc registered [ 3597.318909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3608.531377] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3608.570639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3610.311782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3616.195531] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 3620.833980] LustreError: 104308:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3620.851356] LustreError: 104308:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3621.857649] LustreError: 104309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3626.977685] LustreError: 104308:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3627.206899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3627.607370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3633.333571] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 3643.652714] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3650.860871] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3660.118144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3660.630677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3668.017787] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 3678.512684] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3681.197885] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3696.612753] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3696.643717] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3696.651715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3696.664673] Lustre: Skipped 1 previous similar message [ 3700.638627] Lustre: server umount lustre-OST0000 complete [ 3706.857984] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3706.868444] Lustre: Skipped 1 previous similar message [ 3706.874731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3711.873915] Lustre: server umount lustre-MDT0000 complete [ 3711.973634] LustreError: 105250:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3712.000142] LustreError: 105250:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3715.951173] LustreError: 105045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787821675 with bad export cookie 9605493875812871382 [ 3715.952974] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3715.973878] LustreError: 105045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3716.543823] Lustre: server umount lustre-MDT0001 complete [ 3730.644346] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 3734.891869] Key type lgssc unregistered [ 3735.260672] LNet: 107868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3735.276144] LNetError: 107868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3735.298429] LNet: Removed LNI 192.168.204.104@tcp [ 3736.537105] Key type .llcrypt unregistered [ 3736.541696] Key type ._llcrypt unregistered [ 3760.785633] Key type ._llcrypt registered [ 3760.792418] Key type .llcrypt registered [ 3760.945268] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 3762.677247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3762.986953] alg: No test for adler32 (adler32-zlib) [ 3764.158424] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 3764.403091] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 3766.095408] Key type lgssc registered [ 3767.712531] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3777.411615] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3777.440287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3793.184308] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3798.305972] LustreError: 109113:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.104@tcp: failed processing log, type 4: rc = -110 [ 3824.991578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3832.328106] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 3842.334625] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3850.010913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3850.395964] LustreError: MGC192.168.204.104@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3850.409597] LustreError: 110137:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3850.418883] LustreError: 110137:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 3850.424206] LustreError: 110137:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 3850.527993] Lustre: server umount lustre-MDT0000 complete [ 3850.532583] LustreError: 110137:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3868.128608] Lustre: 108448:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787821812/real 1787821812] req@000000005924c85e x1874666952068480/t0(0) o400->MGC192.168.204.104@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787821828 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3868.189414] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3868.215442] LustreError: 109125:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3888.742335] Lustre: server umount lustre-OST0000 complete [ 3901.537932] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_hostid [ 3913.296049] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 3958.027724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3958.383739] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3958.414256] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3958.548762] Lustre: lustre-MDT0000: new disk, initializing [ 3958.639080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3958.665681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3964.507106] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 3976.285772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3976.476146] Lustre: 112933:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3976.572647] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3976.576651] Lustre: Skipped 1 previous similar message [ 3976.677385] Lustre: lustre-MDT0001: new disk, initializing [ 3976.836962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3976.998298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3977.002386] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3982.818818] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 3993.682599] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4000.386981] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4007.898590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4008.277970] Lustre: lustre-OST0000: new disk, initializing [ 4008.296074] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4008.305603] Lustre: 114913:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4008.397472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4015.006456] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4016.739535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4016.757512] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4016.922669] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4025.942660] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4030.669257] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4030.896981] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4035.581080] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4035.770397] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4042.726940] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4042.728994] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4042.738180] LustreError: Skipped 1 previous similar message [ 4042.741839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4042.754942] Lustre: Skipped 1 previous similar message [ 4042.789716] Lustre: Skipped 1 previous similar message [ 4043.875777] Lustre: server umount lustre-OST0000 complete [ 4047.843364] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4047.852552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4047.857335] Lustre: Skipped 1 previous similar message [ 4047.872495] Lustre: Skipped 1 previous similar message [ 4052.966587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4052.973988] Lustre: Skipped 1 previous similar message [ 4054.150565] Lustre: server umount lustre-MDT0000 complete [ 4057.704854] LustreError: 112927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787822017 with bad export cookie 17667557345822422500 [ 4057.708681] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4057.717909] LustreError: 112927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4058.011583] Lustre: server umount lustre-MDT0001 complete [ 4066.595466] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 05:13:44 (1787822024) [ 4069.699994] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 4086.173781] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_hostid [ 4098.527839] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 4141.477284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4141.808243] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4141.866682] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4141.956487] Lustre: lustre-MDT0000: new disk, initializing [ 4142.070151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4142.092637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4147.830872] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4160.472164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4160.661150] Lustre: 119813:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4160.727590] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4160.731813] Lustre: Skipped 1 previous similar message [ 4160.814929] Lustre: lustre-MDT0001: new disk, initializing [ 4160.912867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4160.948391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4160.968298] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4166.101735] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4176.926086] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4182.930861] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4189.847143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4190.125257] Lustre: lustre-OST0000: new disk, initializing [ 4190.128625] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4190.142317] Lustre: 121781:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4190.246169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4196.426528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4196.438169] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4196.598443] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4196.660892] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4206.712250] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4211.962775] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4212.177388] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4216.884992] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4217.120341] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4222.439954] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4222.441576] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4222.455964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4222.464053] LustreError: Skipped 1 previous similar message [ 4227.573331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4227.595023] Lustre: Skipped 2 previous similar messages [ 4233.183216] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4233.539497] Lustre: server umount lustre-OST0000 complete [ 4237.792334] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4237.796876] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4237.819034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4237.820348] Lustre: Skipped 2 previous similar messages [ 4237.830088] Lustre: Skipped 3 previous similar messages [ 4244.196447] Lustre: server umount lustre-MDT0000 complete [ 4249.070062] LustreError: 121780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787822209 with bad export cookie 17667557345822423382 [ 4249.077342] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4249.088040] LustreError: 121780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4249.488540] Lustre: server umount lustre-MDT0001 complete [ 4260.598285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4261.108228] LustreError: 124003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4261.138522] LustreError: 124003:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4261.207751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4266.478926] LustreError: 124004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4266.557193] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4271.585356] LustreError: 124003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4276.408894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4276.706067] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4276.718977] Lustre: Skipped 2 previous similar messages [ 4282.180867] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4290.544226] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4297.015968] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4304.615283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4312.771377] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4320.989609] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4327.792944] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4334.368582] Lustre: server umount lustre-OST0000 complete [ 4338.150954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4338.155666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4338.169798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4348.443466] Lustre: server umount lustre-MDT0000 complete [ 4352.494879] LustreError: 124003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4352.511294] LustreError: 124003:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4352.537546] LustreError: 123985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787822312 with bad export cookie 17667557345822424264 [ 4352.539419] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4352.558089] LustreError: 123985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4362.999857] Lustre: server umount lustre-MDT0001 complete [ 4374.801681] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 4378.122726] Key type lgssc unregistered [ 4378.543336] LNet: 127639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4378.553754] LNetError: 127639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4378.578814] LNet: Removed LNI 192.168.204.104@tcp [ 4379.713094] Key type .llcrypt unregistered [ 4379.715561] Key type ._llcrypt unregistered [ 4402.203428] Key type ._llcrypt registered [ 4402.207628] Key type .llcrypt registered [ 4402.289307] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_hostid [ 4419.428196] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 4421.151514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4421.208210] alg: No test for adler32 (adler32-zlib) [ 4422.310627] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 4422.568756] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 4424.264586] Key type lgssc registered [ 4425.564985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4466.844871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4466.923044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4468.293297] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4468.337435] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4468.456755] Lustre: lustre-MDT0000: new disk, initializing [ 4468.571921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4468.591371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4473.906670] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4485.486307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4485.651808] Lustre: 130800:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4485.700801] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4485.705053] Lustre: Skipped 1 previous similar message [ 4485.775360] Lustre: lustre-MDT0001: new disk, initializing [ 4485.850024] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4485.912192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4485.927394] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4490.578553] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4501.162478] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4508.081493] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4515.982033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4516.425368] Lustre: lustre-OST0000: new disk, initializing [ 4516.432868] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4516.440574] Lustre: 132765:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4516.597113] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4522.079526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4522.095761] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4522.248529] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4523.576455] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4534.247317] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4539.386407] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4539.605861] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4544.469267] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4544.719590] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4548.068152] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4548.070658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4548.074501] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4548.099508] Lustre: Skipped 1 previous similar message [ 4553.184958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4553.196601] Lustre: Skipped 1 previous similar message [ 4558.306270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4558.315492] Lustre: Skipped 1 previous similar message [ 4561.375482] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4561.538364] Lustre: server umount lustre-OST0000 complete [ 4568.033191] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4568.048228] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4568.061965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4568.067425] Lustre: Skipped 1 previous similar message [ 4573.669837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4573.676688] Lustre: Skipped 2 previous similar messages [ 4580.320803] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4580.537196] Lustre: server umount lustre-MDT0000 complete [ 4584.973404] LustreError: 130791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787822544 with bad export cookie 488780353788480550 [ 4584.989376] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4584.997561] LustreError: 130791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4585.582847] Lustre: server umount lustre-MDT0001 complete [ 4595.061951] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 05:22:32 (1787822552) [ 4603.294528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4603.749162] LustreError: 135181:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4603.767115] LustreError: 135181:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4603.811622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4608.995217] LustreError: 135182:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4609.217147] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4613.092541] LustreError: 135181:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4618.218996] LustreError: 135182:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4620.657393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4621.285883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4628.287888] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4637.830546] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4644.048907] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4645.871381] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4645.872325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4645.880203] Lustre: Skipped 2 previous similar messages [ 4645.887183] Lustre: Skipped 3 previous similar messages [ 4660.192077] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4660.371193] Lustre: server umount lustre-MDT0000 complete [ 4662.248754] LustreError: 135182:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4662.274430] LustreError: 135182:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4664.154448] LustreError: 135162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787822624 with bad export cookie 488780353788481432 [ 4664.163617] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4664.172193] LustreError: 135162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4664.514501] Lustre: server umount lustre-MDT0001 complete [ 4672.341647] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 05:23:50 (1787822630) [ 4679.860415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4695.583969] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4700.703892] LustreError: 137599:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.104@tcp: failed processing log, type 4: rc = -110 [ 4726.495316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4733.023822] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4741.773729] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4744.065543] Lustre: server umount lustre-OST0000 complete [ 4753.163815] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 05:25:11 (1787822711) [ 4760.569050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4761.182877] LustreError: 139046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4761.301544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4765.820737] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4774.443403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4780.545052] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4789.928374] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4796.373765] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4804.312572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4804.698831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4804.709706] Lustre: Skipped 1 previous similar message [ 4812.419693] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4820.214110] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4831.718553] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4831.733013] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4831.735320] Lustre: Skipped 1 previous similar message [ 4831.755773] Lustre: Skipped 8 previous similar messages [ 4836.369067] Lustre: server umount lustre-MDT0000 complete [ 4836.835685] LustreError: 139043:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4836.863662] LustreError: 139043:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4839.683719] LustreError: 139028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787822799 with bad export cookie 488780353788482097 [ 4839.695075] LustreError: 139028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4839.695198] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4839.992952] Lustre: server umount lustre-MDT0001 complete [ 4854.136191] Lustre: server umount lustre-OST0000 complete [ 4861.665650] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 05:26:59 (1787822819) [ 4868.979409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4869.532471] LustreError: 142702:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4869.653321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4874.279987] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4882.867204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4888.330897] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4896.157546] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4902.058982] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4908.571843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4914.425897] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 4914.991069] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4921.829959] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4925.986365] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4926.224889] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4931.310185] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4931.574304] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4935.650253] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4935.650667] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4935.661105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4935.664255] Lustre: Skipped 2 previous similar messages [ 4940.104477] Lustre: server umount lustre-OST0000 complete [ 4944.864611] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4944.889370] LustreError: Skipped 1 previous similar message [ 4944.899847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4949.865759] Lustre: server umount lustre-MDT0000 complete [ 4951.009185] LustreError: 142702:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4951.024503] LustreError: 142702:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4953.860836] LustreError: 145812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787822913 with bad export cookie 488780353788483161 [ 4953.862346] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4953.870355] LustreError: 145812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4954.142415] Lustre: server umount lustre-MDT0001 complete [ 4962.809248] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 05:28:40 (1787822920) [ 4969.224157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4984.799627] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4989.920626] LustreError: 146859:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.104@tcp: failed processing log, type 4: rc = -110 [ 5015.647573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5015.681496] Lustre: Skipped 2 previous similar messages [ 5023.095163] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5032.351499] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5039.680986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5040.292496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 5044.366494] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5053.715482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5058.744323] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5065.899516] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5070.583187] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5074.760968] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5074.942529] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5079.476543] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5079.746607] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5085.161845] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5085.167471] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5085.170642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5085.170655] Lustre: Skipped 4 previous similar messages [ 5085.177413] Lustre: Skipped 3 previous similar messages [ 5088.231134] Lustre: server umount lustre-MDT0000 complete [ 5090.285959] LustreError: 147923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5090.334463] LustreError: 147923:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 5092.800339] LustreError: 146867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787823052 with bad export cookie 488780353788484190 [ 5092.809857] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5092.826208] LustreError: 146867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5093.161879] Lustre: server umount lustre-MDT0001 complete [ 5107.866189] Lustre: server umount lustre-OST0000 complete [ 5115.182800] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 05:31:13 (1787823073) [ 5121.296961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5136.927714] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5142.051814] LustreError: 151054:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.104@tcp: failed processing log, type 4: rc = -110 [ 5173.660612] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5181.320212] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5188.545113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5189.243082] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 5193.447552] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5200.618473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5204.529536] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5210.532750] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5214.616650] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5219.884896] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5220.042484] Lustre: lustre-OST0001: new disk, initializing [ 5220.046575] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5220.052746] Lustre: 153929:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5225.742144] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5229.125703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5229.144407] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5229.283849] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5234.713559] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5239.299217] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5239.519659] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5242.712460] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5242.889758] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5244.908311] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5244.926214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5244.930836] Lustre: Skipped 3 previous similar messages [ 5250.523437] Lustre: server umount lustre-OST0000 complete [ 5255.153847] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5260.295877] Lustre: server umount lustre-OST0001 complete [ 5269.559212] Lustre: server umount lustre-MDT0000 complete [ 5272.628029] LustreError: 151062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787823232 with bad export cookie 488780353788485142 [ 5272.637803] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5272.639469] LustreError: 151062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5272.926299] Lustre: server umount lustre-MDT0001 complete [ 5296.290221] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 05:34:13 (1787823253) [ 5298.721227] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5300.925155] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5302.819283] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 05:34:20 (1787823260) [ 5310.536570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5310.985735] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5311.023699] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5311.356127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5311.363813] Lustre: Skipped 6 previous similar messages [ 5315.679659] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5324.240608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5324.409187] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5324.465704] Lustre: 157488:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5330.071983] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5337.980728] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5344.816489] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5353.619244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5353.896212] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5360.837276] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5361.235745] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5369.341072] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5373.795580] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5374.039290] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5378.850284] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5379.099366] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5381.090808] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5381.100683] LustreError: Skipped 1 previous similar message [ 5381.106652] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5381.124908] Lustre: Skipped 5 previous similar messages [ 5387.235701] Lustre: server umount lustre-OST0000 complete [ 5408.294630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5411.816637] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5411.842726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5411.843450] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5411.882377] Lustre: Skipped 1 previous similar message [ 5414.966212] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5422.604402] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5428.586547] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5428.828153] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5433.617790] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5433.832635] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5439.707757] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9f41511cf000.ost_server_uuid 50 [ 5441.420791] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9f41511cf000.ost_server_uuid in FULL state after 0 sec [ 5441.653318] LustreError: 157495:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.204.4@tcp) returned error from blocking AST (req@0000000010691183 x1874667642287744 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000b6f55665/0x6c87f2460c0a770 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.4@tcp remote: 0x3834b780cc72b5ce expref: 5 pid: 157498 timeout: 5541 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5441.703498] LustreError: lustre-MDT0000: A client on nid 192.168.204.4@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5441.714565] LustreError: 157486:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.4@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000b6f55665/0x6c87f2460c0a770 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.4@tcp remote: 0x3834b780cc72b5ce expref: 6 pid: 157498 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5447.661789] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5452.942670] Lustre: server umount lustre-OST0000 complete [ 5463.279500] Lustre: server umount lustre-MDT0000 complete [ 5466.473403] LustreError: 157482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787823426 with bad export cookie 488780353788486325 [ 5466.479291] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5466.483611] LustreError: 157482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5472.806846] Lustre: server umount lustre-MDT0001 complete [ 5482.176117] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 5485.782219] Key type lgssc unregistered [ 5486.078695] LNet: 163610:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5486.091634] LNetError: 163610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5486.139876] LNet: Removed LNI 192.168.204.104@tcp [ 5487.414664] Key type .llcrypt unregistered [ 5487.418785] Key type ._llcrypt unregistered [ 5513.675837] Key type ._llcrypt registered [ 5513.678846] Key type .llcrypt registered [ 5513.818047] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 5515.028380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5515.084097] alg: No test for adler32 (adler32-zlib) [ 5516.151203] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 5516.459504] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 5518.175688] Key type lgssc registered [ 5519.210404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5526.332890] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5526.346475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5527.822538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5532.478916] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5537.254706] LustreError: 165025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5537.267848] LustreError: 165025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5540.781064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5541.286672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5546.198404] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5553.299892] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5558.509278] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5565.265493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5565.611070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5571.599039] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5572.783745] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5578.761328] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5585.086193] Lustre: Failing over lustre-MDT0000 [ 5585.378237] Lustre: server umount lustre-MDT0000 complete [ 5586.654674] LustreError: 165022:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5587.429606] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5587.446148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5588.450790] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5588.451463] LustreError: 165025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5588.477337] LustreError: 165025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5593.569038] LustreError: 165026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5593.591551] LustreError: 165026:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5597.395818] LustreError: 165007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.4@tcp arrived at 1787823557 with bad export cookie 3661853388407932933 [ 5597.427867] LustreError: 165007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5598.691214] LustreError: 165753:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5598.720040] LustreError: 165753:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5604.383540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5604.524930] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5604.733477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5604.799175] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5609.109332] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5609.957791] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5609.963293] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5611.375033] Lustre: lustre-MDT0000: Denying connection for new client 16c192c4-ee1b-41c0-8cdd-7f9a52811490 (at 192.168.204.4@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 5620.195073] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5620.203514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5620.215389] Lustre: Skipped 2 previous similar messages [ 5625.297664] Lustre: server umount lustre-OST0000 complete [ 5628.649633] LustreError: 168753:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5628.658345] Lustre: 168100:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5628.663428] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5628.679797] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5628.689254] LustreError: 168100:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000eeeea568 x1874668789214592/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5628.704842] LustreError: 168100:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5628.717191] LustreError: 168100:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5628.726603] Lustre: lustre-MDT0000: Recovery over after 0:19, of 2 clients 0 recovered and 2 were evicted. [ 5629.012853] Lustre: server umount lustre-MDT0000 complete [ 5630.433570] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5630.462305] LustreError: 165021:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5630.476150] LustreError: 165021:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5632.282311] LustreError: 168464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787823592 with bad export cookie 3661853388407933031 [ 5632.289834] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5632.295609] LustreError: 168464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5634.587184] Lustre: server umount lustre-MDT0001 complete [ 5644.225670] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 5647.038190] Key type lgssc unregistered [ 5647.332976] LNet: 169507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5647.342098] LNetError: 169507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5647.360813] LNet: Removed LNI 192.168.204.104@tcp [ 5648.269518] Key type .llcrypt unregistered [ 5648.273345] Key type ._llcrypt unregistered [ 5672.438763] Key type ._llcrypt registered [ 5672.441612] Key type .llcrypt registered [ 5672.615703] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 5674.021596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5674.096821] alg: No test for adler32 (adler32-zlib) [ 5675.382517] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 5675.806111] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 5677.575191] Key type lgssc registered [ 5679.205893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5687.441754] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5687.465871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5688.768510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5693.407833] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5698.023087] LustreError: 170922:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5698.038046] LustreError: 170922:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5702.898139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5703.231110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5710.019079] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5718.395650] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5724.632846] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5732.506888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5733.051365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5734.085468] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5740.423121] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5747.813317] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5753.838895] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5753.852318] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5753.858803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5753.871355] Lustre: Skipped 1 previous similar message [ 5758.951069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5758.960256] Lustre: Skipped 2 previous similar messages [ 5759.751928] Lustre: server umount lustre-OST0000 complete [ 5764.065316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5764.068575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5764.082246] Lustre: Skipped 1 previous similar message [ 5764.097879] Lustre: Skipped 1 previous similar message [ 5769.188595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5769.195510] Lustre: Skipped 1 previous similar message [ 5769.483685] Lustre: server umount lustre-MDT0000 complete [ 5772.919752] LustreError: 170902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787823732 with bad export cookie 5320777523186413306 [ 5772.923946] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5772.931907] LustreError: 170902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5773.433663] Lustre: server umount lustre-MDT0001 complete [ 5782.980038] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 5785.744412] Key type lgssc unregistered [ 5786.008991] LNet: 174458:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5786.018261] LNetError: 174458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5786.045734] LNet: Removed LNI 192.168.204.104@tcp [ 5786.805233] Key type .llcrypt unregistered [ 5786.807994] Key type ._llcrypt unregistered [ 5817.063891] Key type ._llcrypt registered [ 5817.066937] Key type .llcrypt registered [ 5817.160299] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 5818.137183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5818.211153] alg: No test for adler32 (adler32-zlib) [ 5819.298613] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 5819.499813] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 5821.135167] Key type lgssc registered [ 5822.184561] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5829.328614] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5829.349607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5830.731797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5835.227384] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5840.866239] LustreError: 176166:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5840.894641] LustreError: 176166:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5843.019892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5843.429759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5847.883936] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5854.925766] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5860.977424] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5868.528330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5868.973732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5875.197406] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5875.617684] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5883.875760] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5890.536025] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5890.545888] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5890.560095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5892.652883] Lustre: server umount lustre-OST0000 complete [ 5899.744219] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5899.753329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5899.764525] Lustre: Skipped 1 previous similar message [ 5899.774385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5899.792540] Lustre: Skipped 1 previous similar message [ 5900.776194] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5900.797177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5900.803342] Lustre: Skipped 1 previous similar message [ 5902.754467] Lustre: server umount lustre-MDT0000 complete [ 5905.890237] LustreError: 177189:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5905.929995] LustreError: 177189:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5906.599522] LustreError: 178947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787823866 with bad export cookie 3671733887292524639 [ 5906.603200] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5906.608913] LustreError: 178947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5907.121040] Lustre: server umount lustre-MDT0001 complete [ 5917.954946] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 5920.626024] Key type lgssc unregistered [ 5920.958197] LNet: 179701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5920.970972] LNetError: 179701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5921.003323] LNet: Removed LNI 192.168.204.104@tcp [ 5921.973721] Key type .llcrypt unregistered [ 5921.981920] Key type ._llcrypt unregistered [ 5943.127740] Key type ._llcrypt registered [ 5943.130436] Key type .llcrypt registered [ 5943.278854] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 5944.340248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5944.431683] alg: No test for adler32 (adler32-zlib) [ 5945.565404] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 5945.857607] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 5947.575325] Key type lgssc registered [ 5948.582906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5959.778078] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5959.799724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5961.145221] Lustre: *** cfs_fail_loc=135, val=0*** [ 5961.266433] LustreError: 181226:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5961.271610] LustreError: 181226:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.104@tcp: cfg command failed: rc = -2 [ 5961.282511] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5961.282511] [ 5961.292944] LustreError: MGC192.168.204.104@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5961.306323] LustreError: 181187:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5961.324739] LustreError: 181187:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 5961.338415] LustreError: 181187:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 5961.349904] LustreError: 181187:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5961.555889] Lustre: server umount lustre-MDT0000 complete [ 5961.564793] LustreError: 181187:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5968.370868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5968.948633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5973.364896] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5978.085883] LustreError: 181597:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5978.102516] LustreError: 181597:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5981.675826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5982.037570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5986.564366] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 5994.236897] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5999.489668] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6002.657591] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6002.667298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6002.683927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6003.680176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6003.681874] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6006.818027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6006.827158] Lustre: Skipped 2 previous similar messages [ 6007.990862] Lustre: server umount lustre-MDT0000 complete [ 6011.555370] LustreError: 181578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787823971 with bad export cookie 18283811977722096489 [ 6011.556677] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6011.562141] LustreError: 181578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6011.904047] Lustre: server umount lustre-MDT0001 complete [ 6022.078263] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 6024.814464] Key type lgssc unregistered [ 6025.125658] LNet: 183925:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6025.130183] LNetError: 183925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6025.149216] LNet: Removed LNI 192.168.204.104@tcp [ 6025.993738] Key type .llcrypt unregistered [ 6025.996191] Key type ._llcrypt unregistered [ 6047.960675] Key type ._llcrypt registered [ 6047.964361] Key type .llcrypt registered [ 6048.078077] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 6048.508329] Key type .llcrypt unregistered [ 6048.513638] Key type ._llcrypt unregistered [ 6067.578792] Key type ._llcrypt registered [ 6067.582695] Key type .llcrypt registered [ 6067.755500] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 6069.011902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6069.296065] alg: No test for adler32 (adler32-zlib) [ 6070.405668] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 6070.702443] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 6072.463220] Key type lgssc registered [ 6073.607279] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6081.577556] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6081.592538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6097.119461] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6102.239910] LustreError: 186063:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.104@tcp: failed processing log, type 4: rc = -110 [ 6128.929571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6135.402549] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6142.791502] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6150.108058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6150.527801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6150.611312] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6155.028585] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6155.756135] LustreError: 186088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6155.775877] LustreError: 186088:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6160.875553] LustreError: 186089:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6163.787355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6163.958509] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6163.966990] Lustre: Skipped 1 previous similar message [ 6164.187352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6168.802980] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6175.955552] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6181.030299] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6183.841248] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6196.707285] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6196.712154] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6196.730358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6200.290726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6200.294579] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6202.764728] Lustre: server umount lustre-OST0000 complete [ 6210.530401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6210.534063] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6210.557073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6210.557081] Lustre: Skipped 1 previous similar message [ 6212.474361] Lustre: server umount lustre-MDT0000 complete [ 6215.650038] LustreError: 187104:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6215.676631] LustreError: 187104:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6216.102434] LustreError: 186069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787824176 with bad export cookie 445841742755204437 [ 6216.117384] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6216.118564] LustreError: 186069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6216.574263] Lustre: server umount lustre-MDT0001 complete [ 6226.742619] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 6229.720508] Key type lgssc unregistered [ 6230.110835] LNet: 190080:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6230.115299] LNetError: 190080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6230.129160] LNet: Removed LNI 192.168.204.104@tcp [ 6231.021082] Key type .llcrypt unregistered [ 6231.023888] Key type ._llcrypt unregistered [ 6254.166932] Key type ._llcrypt registered [ 6254.168936] Key type .llcrypt registered [ 6254.251862] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 6255.101115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6255.195080] alg: No test for adler32 (adler32-zlib) [ 6256.400253] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 6256.709554] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 6258.383274] Key type lgssc registered [ 6259.507091] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6268.032919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6268.062730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6269.542414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6274.606076] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6279.140388] LustreError: 191487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6279.160189] LustreError: 191487:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6282.825635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6283.210865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6287.723279] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6295.295896] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6301.763681] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6309.218978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6309.634768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6314.818264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6316.050989] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6324.529251] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6330.154880] Lustre: Failing over lustre-MDT0000 [ 6330.506332] Lustre: server umount lustre-MDT0000 complete [ 6331.652932] LustreError: 191481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6333.411971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6333.414642] LustreError: 191482:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6333.438030] Lustre: Skipped 2 previous similar messages [ 6333.472573] LustreError: 191482:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6336.709758] LustreError: 193277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6341.835159] LustreError: 191482:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6341.860629] LustreError: 191482:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6349.793770] Lustre: 190796:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787824293/real 1787824293] req@000000009623caef x1874669565142912/t0(0) o400->MGC192.168.204.104@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787824309 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6349.830148] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6349.864468] LustreError: 192215:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6349.892472] LustreError: 192215:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6350.725463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6359.321896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6359.401603] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6362.316858] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6364.665606] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6364.685532] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 6364.723506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6365.016661] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6369.913969] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6379.238971] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6393.320443] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6393.359368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6393.371087] Lustre: Skipped 1 previous similar message [ 6398.443917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6398.448704] Lustre: Skipped 1 previous similar message [ 6398.736420] Lustre: server umount lustre-OST0000 complete [ 6403.555134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6403.568045] Lustre: Skipped 1 previous similar message [ 6403.580740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6403.588804] Lustre: Skipped 1 previous similar message [ 6408.673992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6408.683229] Lustre: Skipped 1 previous similar message [ 6409.674191] Lustre: server umount lustre-MDT0000 complete [ 6413.792593] LustreError: 191483:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6413.824162] LustreError: 191483:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6413.972714] LustreError: 191468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787824373 with bad export cookie 3861085418141234665 [ 6413.978327] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6413.984668] LustreError: 191468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6414.378860] Lustre: server umount lustre-MDT0001 complete [ 6426.756543] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 6429.706571] Key type lgssc unregistered [ 6429.988023] LNet: 196405:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6429.995511] LNetError: 196405:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6430.029457] LNet: Removed LNI 192.168.204.104@tcp [ 6430.818709] Key type .llcrypt unregistered [ 6430.820907] Key type ._llcrypt unregistered [ 6454.035910] Key type ._llcrypt registered [ 6454.038626] Key type .llcrypt registered [ 6454.148676] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 6455.382945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6455.409309] alg: No test for adler32 (adler32-zlib) [ 6456.431875] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 6456.710571] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 6458.375229] Key type lgssc registered [ 6459.366177] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6467.591355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6467.604348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6469.223852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6473.801823] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6479.858616] LustreError: 197818:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6479.881239] LustreError: 197818:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6483.547977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6484.042162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6488.732290] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6497.497347] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6504.735134] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6512.586241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6513.071748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6514.100818] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6519.181888] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6527.493965] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6531.465989] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6535.232425] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6549.998155] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6564.322558] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6564.336777] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6564.344697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6564.388691] Lustre: Skipped 1 previous similar message [ 6567.988994] Lustre: server umount lustre-OST0000 complete [ 6574.565569] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6574.582704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6574.601522] Lustre: Skipped 1 previous similar message [ 6574.621341] Lustre: Skipped 1 previous similar message [ 6579.554958] Lustre: server umount lustre-MDT0000 complete [ 6579.682439] LustreError: 198546:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6579.706597] LustreError: 198546:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6584.807200] LustreError: 197812:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6584.835637] LustreError: 197812:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6585.595262] LustreError: 197797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787824545 with bad export cookie 112933513346254393 [ 6585.606333] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6586.175932] Lustre: server umount lustre-MDT0001 complete [ 6598.654930] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 6601.752940] Key type lgssc unregistered [ 6602.051747] LNet: 201670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6602.068535] LNetError: 201670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6602.101914] LNet: Removed LNI 192.168.204.104@tcp [ 6603.335675] Key type .llcrypt unregistered [ 6603.340759] Key type ._llcrypt unregistered [ 6625.213402] Key type ._llcrypt registered [ 6625.215890] Key type .llcrypt registered [ 6625.328686] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 6626.470900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6626.528262] alg: No test for adler32 (adler32-zlib) [ 6627.867708] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 6628.184370] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 6629.887196] Key type lgssc registered [ 6631.364639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6639.401062] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6639.437409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6640.955899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6646.272151] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6650.346842] LustreError: 202940:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6650.381235] LustreError: 202940:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6655.458796] LustreError: 202941:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6656.049463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6656.420802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6661.808621] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6669.561399] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6674.917409] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6681.131282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6681.452332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6685.554743] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6687.519292] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6695.191517] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6702.209144] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6716.311625] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6723.335807] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6732.034934] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6741.985637] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6741.986481] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6742.005954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6742.026887] Lustre: Skipped 1 previous similar message [ 6747.109334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6747.116271] Lustre: Skipped 1 previous similar message [ 6752.230514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6752.235531] Lustre: Skipped 1 previous similar message [ 6756.319451] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6756.565458] Lustre: server umount lustre-OST0000 complete [ 6762.467475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6762.485797] Lustre: Skipped 1 previous similar message [ 6762.495526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6762.502480] Lustre: Skipped 1 previous similar message [ 6767.586275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6767.601418] Lustre: Skipped 1 previous similar message [ 6775.265689] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6775.496043] Lustre: server umount lustre-MDT0000 complete [ 6777.827635] LustreError: 202936:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6777.862119] LustreError: 202936:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6779.408094] LustreError: 202922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787824739 with bad export cookie 1760810845333321484 [ 6779.410911] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6779.429771] LustreError: 202922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6779.853646] Lustre: server umount lustre-MDT0001 complete [ 6790.278444] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 6792.898875] Key type lgssc unregistered [ 6793.245418] LNet: 207825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6793.254222] LNetError: 207825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6793.274917] LNet: Removed LNI 192.168.204.104@tcp [ 6794.188684] Key type .llcrypt unregistered [ 6794.194543] Key type ._llcrypt unregistered [ 6817.204673] Key type ._llcrypt registered [ 6817.207772] Key type .llcrypt registered [ 6817.334844] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 6818.551726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6818.766175] alg: No test for adler32 (adler32-zlib) [ 6819.897519] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 6820.254074] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 6821.911342] Key type lgssc registered [ 6823.013441] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6831.111216] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6831.141129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6832.546675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6836.397594] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6842.862534] LustreError: 209255:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6842.898486] LustreError: 209255:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6844.550046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6845.036466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6849.226799] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6856.118462] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6862.576265] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6869.838021] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6869.992616] LustreError: 211076:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6870.385528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6876.620804] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6877.699190] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6884.476160] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6901.217029] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6901.232677] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6901.244168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6903.265284] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6903.271312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6903.286616] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6903.301852] Lustre: Skipped 1 previous similar message [ 6905.594207] Lustre: server umount lustre-OST0000 complete [ 6911.456319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6911.464410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6911.483910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6913.511536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6913.512317] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6918.624020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6918.628597] Lustre: Skipped 2 previous similar messages [ 6924.256045] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6924.508453] Lustre: server umount lustre-MDT0000 complete [ 6928.337390] LustreError: 209235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787824888 with bad export cookie 16927709373508237126 [ 6928.367240] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6928.371766] LustreError: 209235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6928.812325] Lustre: server umount lustre-MDT0001 complete [ 6940.108528] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 6943.077769] Key type lgssc unregistered [ 6943.472624] LNet: 212952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6943.482841] LNetError: 212952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6943.506816] LNet: Removed LNI 192.168.204.104@tcp [ 6944.369453] Key type .llcrypt unregistered [ 6944.372620] Key type ._llcrypt unregistered [ 6961.842737] Key type ._llcrypt registered [ 6961.846602] Key type .llcrypt registered [ 6961.974878] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 6963.135775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6963.166858] alg: No test for adler32 (adler32-zlib) [ 6964.259872] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 6964.482756] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 6966.199439] Key type lgssc registered [ 6967.626739] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6977.080807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6977.109108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6978.779864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6983.408962] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 6988.276892] LustreError: 214199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6988.295722] LustreError: 214199:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6992.164372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6992.519410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6996.593707] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 7004.330253] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7012.093347] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7020.038816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7020.488807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7023.535995] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 7027.375951] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 7035.843688] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7053.306213] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7053.309119] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7053.317834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7053.328992] Lustre: Skipped 1 previous similar message [ 7057.411611] Lustre: server umount lustre-OST0000 complete [ 7063.521657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7063.522438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7063.549986] Lustre: Skipped 2 previous similar messages [ 7067.867922] Lustre: server umount lustre-MDT0000 complete [ 7068.643145] LustreError: 214929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7068.670604] LustreError: 214929:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7072.378632] LustreError: 214179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787825032 with bad export cookie 15165093002092915842 [ 7072.389730] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7072.390021] LustreError: 214179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7072.760404] Lustre: server umount lustre-MDT0001 complete [ 7086.060836] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 7089.231915] Key type lgssc unregistered [ 7089.667837] LNet: 217888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7089.680739] LNetError: 217888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7089.741221] LNet: Removed LNI 192.168.204.104@tcp [ 7090.711688] Key type .llcrypt unregistered [ 7090.713947] Key type ._llcrypt unregistered [ 7112.832973] Key type ._llcrypt registered [ 7112.835075] Key type .llcrypt registered [ 7112.938861] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 7114.631467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7114.673404] alg: No test for adler32 (adler32-zlib) [ 7115.812376] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 7116.108471] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 7117.807470] Key type lgssc registered [ 7118.985597] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7128.443169] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7128.485205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7130.021317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7135.188529] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 7139.302553] LustreError: 219155:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7139.346510] LustreError: 219155:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7144.418659] LustreError: 219154:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7144.976483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7145.427489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7150.981740] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 7160.268942] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7166.542244] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7174.342146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7174.738750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7181.814356] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 7191.675638] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7199.344472] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7213.050736] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7213.180304] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7213.498986] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7221.214388] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 7230.399732] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7232.304949] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7244.633909] Lustre: Permanently deactivating lustre-OST0001 [ 7244.669801] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7254.593896] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7262.910537] Lustre: Permanently reactivating lustre-OST0001 [ 7262.927528] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7262.934336] Lustre: Skipped 2 previous similar messages [ 7269.555147] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7269.575169] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7269.585817] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7269.609922] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7269.644835] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7274.977983] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7274.984259] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7275.002807] LustreError: Skipped 1 previous similar message [ 7275.004929] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7275.048118] Lustre: Skipped 2 previous similar messages [ 7276.015201] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7276.042336] Lustre: Skipped 2 previous similar messages [ 7279.086135] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7279.101684] Lustre: Skipped 1 previous similar message [ 7284.197709] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7286.239135] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7286.434830] Lustre: server umount lustre-OST0001 complete [ 7289.317306] LustreError: 220977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7289.353583] LustreError: 220977:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7291.875138] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7291.898260] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7291.916084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7291.927417] Lustre: Skipped 1 previous similar message [ 7292.909121] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7294.441593] LustreError: 222497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7294.455242] LustreError: 222497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7299.582338] LustreError: 220977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7299.622941] LustreError: 220977:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7304.685891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7304.695510] Lustre: Skipped 6 previous similar messages [ 7305.183917] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7305.510279] Lustre: server umount lustre-OST0000 complete [ 7314.400735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7314.421140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7314.452679] Lustre: Skipped 1 previous similar message [ 7325.152219] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7325.153177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7325.173729] Lustre: Skipped 5 previous similar messages [ 7325.357569] Lustre: server umount lustre-MDT0000 complete [ 7329.315947] LustreError: 219135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787825289 with bad export cookie 8642885468607944564 [ 7329.336877] LustreError: 219135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7329.341953] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7329.894987] Lustre: server umount lustre-MDT0001 complete [ 7341.981623] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 7346.121895] Key type lgssc unregistered [ 7346.597724] LNet: 224252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7346.613909] LNetError: 224252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7346.646619] LNet: Removed LNI 192.168.204.104@tcp [ 7347.482341] Key type .llcrypt unregistered [ 7347.484561] Key type ._llcrypt unregistered [ 7398.512718] Key type ._llcrypt registered [ 7398.516553] Key type .llcrypt registered [ 7398.682282] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 7400.086224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7400.150192] alg: No test for adler32 (adler32-zlib) [ 7401.284571] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 7401.470740] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 7403.095397] Key type lgssc registered [ 7404.504459] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7414.072832] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7414.086372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7415.470335] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7415.500514] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7415.713350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7421.229977] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 7432.350167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7432.545898] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7432.598054] Lustre: 226458:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7432.974230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7438.218729] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 7448.310540] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7455.755471] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7464.780501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7464.998914] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7465.239388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7472.702422] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 7473.938780] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7483.233042] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7487.406699] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7492.579925] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7501.044584] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7511.624593] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7523.543928] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7523.548343] Lustre: Skipped 1 previous similar message [ 7547.669857] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7547.675600] Lustre: Skipped 2 previous similar messages [ 7579.743845] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7579.754377] Lustre: Skipped 3 previous similar messages [ 7655.101248] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 7655.110861] Lustre: Skipped 8 previous similar messages [ 7668.705552] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7668.717958] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7668.731553] LustreError: Skipped 1 previous similar message [ 7668.735314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7668.745626] Lustre: Skipped 1 previous similar message [ 7668.765863] Lustre: Skipped 1 previous similar message [ 7670.552557] Lustre: server umount lustre-OST0000 complete [ 7678.958353] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7678.977942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7678.979927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7678.987262] Lustre: Skipped 1 previous similar message [ 7679.013772] Lustre: Skipped 1 previous similar message [ 7681.756186] Lustre: server umount lustre-MDT0000 complete [ 7686.665044] LustreError: 226451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787825646 with bad export cookie 522136390225230637 [ 7686.666939] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7686.679840] LustreError: 226451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7687.210894] Lustre: server umount lustre-MDT0001 complete [ 7700.993806] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 7703.723972] Key type lgssc unregistered [ 7704.053283] LNet: 231272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7704.065251] LNetError: 231272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7704.084619] LNet: Removed LNI 192.168.204.104@tcp [ 7705.142153] Key type .llcrypt unregistered [ 7705.144848] Key type ._llcrypt unregistered [ 7733.330187] Key type ._llcrypt registered [ 7733.332156] Key type .llcrypt registered [ 7733.442501] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 7735.225045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7735.256048] alg: No test for adler32 (adler32-zlib) [ 7736.385361] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 7736.630294] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 7738.279822] Key type lgssc registered [ 7739.893133] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7748.604287] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7748.648206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7750.191336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7755.154504] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 7760.354107] LustreError: 232685:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7760.378670] LustreError: 232685:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7764.182341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7764.618379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7769.384879] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 7779.271222] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7786.198710] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7794.238676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7794.737631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7795.770187] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7801.665215] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_default_debug -1 all [ 7810.837732] Lustre: DEBUG MARKER: oleg404-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7834.598366] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7834.598544] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7834.660370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7834.665460] Lustre: Skipped 1 previous similar message [ 7839.726749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7839.743420] Lustre: Skipped 1 previous similar message [ 7844.835256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7844.841578] Lustre: Skipped 1 previous similar message [ 7847.391240] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7847.562248] Lustre: server umount lustre-OST0000 complete [ 7855.075434] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7855.079130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7855.112670] Lustre: Skipped 2 previous similar messages [ 7855.134931] Lustre: Skipped 1 previous similar message [ 7857.877194] Lustre: server umount lustre-MDT0000 complete [ 7860.193029] LustreError: 232682:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7860.212064] LustreError: 232682:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7862.127223] LustreError: 232666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787825822 with bad export cookie 10269629174421159006 [ 7862.130535] LustreError: MGC192.168.204.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7862.146478] LustreError: 232666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7862.503605] Lustre: server umount lustre-MDT0001 complete [ 7878.076881] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 7881.237680] Key type lgssc unregistered [ 7881.600899] LNet: 236380:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7881.626522] LNetError: 236380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7881.643759] LNet: Removed LNI 192.168.204.104@tcp [ 7882.583809] Key type .llcrypt unregistered [ 7882.591991] Key type ._llcrypt unregistered [ 7910.428170] Key type ._llcrypt registered [ 7910.430863] Key type .llcrypt registered [ 7910.559638] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 7912.137661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7912.217769] alg: No test for adler32 (adler32-zlib) [ 7913.374792] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 7913.655365] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 7915.407211] Key type lgssc registered [ 7916.804921] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8187.208187] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 8190.358675] Key type lgssc unregistered [ 8190.745836] LNet: 238405:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8190.752906] LNetError: 238405:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8190.784875] LNet: Removed LNI 192.168.204.104@tcp [ 8191.963964] Key type .llcrypt unregistered [ 8191.969125] Key type ._llcrypt unregistered [ 8224.354459] Key type ._llcrypt registered [ 8224.356922] Key type .llcrypt registered [ 8224.468209] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing set_hostid [ 8243.461554] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing load_modules_local [ 8245.065652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8245.199370] alg: No test for adler32 (adler32-zlib) [ 8246.407823] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 8246.870785] LNet: Added LNI 192.168.204.104@tcp [8/256/0/180] [ 8248.607206] Key type lgssc registered [ 8250.489982] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8287.392436] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7940 sec ========== 06:24:04 (1787826244) [ 8289.567881] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:24:06 (1787826246) === [ 8293.661544] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:24:11 (1787826251) === [ 8339.961310] Lustre: DEBUG MARKER: oleg404-server.virtnet: executing unload_modules_local [ 8344.927680] Key type lgssc unregistered [ 8345.444830] LNet: 243651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8345.468991] LNetError: 243651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8345.495197] LNet: Removed LNI 192.168.204.104@tcp [ 8346.526535] Key type .llcrypt unregistered [ 8346.529272] Key type ._llcrypt unregistered