[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 496639426 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003226] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008334] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.010019] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.011027] pid_max: default: 32768 minimum: 301 [ 0.013057] LSM: Security Framework initializing [ 0.014067] Yama: becoming mindful. [ 0.015046] SELinux: Initializing. [ 0.016072] *** VALIDATE selinux *** [ 0.025594] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.031784] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.033177] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034136] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.036018] *** VALIDATE tmpfs *** [ 0.037500] *** VALIDATE proc *** [ 0.038301] *** VALIDATE cgroup *** [ 0.039015] *** VALIDATE cgroup2 *** [ 0.040301] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.041169] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.042018] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.043037] Spectre V2 : User space: Vulnerable [ 0.044013] Speculative Store Bypass: Vulnerable [ 0.047421] debug: unmapping init [mem 0xffffffff9c859000-0xffffffff9c860fff] [ 0.049205] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.050784] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.051026] ... version: 2 [ 0.052019] ... bit width: 48 [ 0.053014] ... generic registers: 4 [ 0.054015] ... value mask: 0000ffffffffffff [ 0.055017] ... max period: 00007fffffffffff [ 0.056018] ... fixed-purpose events: 3 [ 0.057015] ... event mask: 000000070000000f [ 0.058319] rcu: Hierarchical SRCU implementation. [ 0.060584] smp: Bringing up secondary CPUs ... [ 0.061659] x86: Booting SMP configuration: [ 0.062037] .... node #0, CPUs: #1 #2 #3 [ 0.066012] smp: Brought up 1 node, 4 CPUs [ 0.068021] smpboot: Max logical packages: 1 [ 0.069015] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.249043] node 0 deferred pages initialised in 176ms [ 0.252265] devtmpfs: initialized [ 0.254343] x86/mm: Memory block size: 128MB [ 0.256993] gcov: version magic: 0x41383552 [ 0.260374] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.264112] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.266466] pinctrl core: initialized pinctrl subsystem [ 0.268212] [ 0.268772] ************************************************************* [ 0.271018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.273016] ** ** [ 0.275017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.277019] ** ** [ 0.279017] ** This means that this kernel is built to expose internal ** [ 0.281021] ** IOMMU data structures, which may compromise security on ** [ 0.284016] ** your system. ** [ 0.286019] ** ** [ 0.288019] ** If you see this message and you are not debugging the ** [ 0.290022] ** kernel, report this immediately to your vendor! ** [ 0.292019] ** ** [ 0.295015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.297016] ************************************************************* [ 0.299859] NET: Registered protocol family 16 [ 0.302567] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.305089] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.307083] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.311153] cpuidle: using governor menu [ 0.312746] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.316716] PCI: Using configuration type 1 for base access [ 0.320135] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.331211] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.334122] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.338038] cryptd: max_cpu_qlen set to 1000 [ 0.342169] ACPI: Added _OSI(Module Device) [ 0.343024] ACPI: Added _OSI(Processor Device) [ 0.344021] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.345018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.348758] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.356507] ACPI: Interpreter enabled [ 0.357089] ACPI: PM: (supports S0 S3 S4 S5) [ 0.359021] ACPI: Using IOAPIC for interrupt routing [ 0.360169] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.364457] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.377823] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.379063] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.382034] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.385112] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.391463] acpiphp: Slot [2] registered [ 0.393137] acpiphp: Slot [5] registered [ 0.394148] acpiphp: Slot [6] registered [ 0.396246] acpiphp: Slot [7] registered [ 0.397166] acpiphp: Slot [8] registered [ 0.399165] acpiphp: Slot [9] registered [ 0.400187] acpiphp: Slot [10] registered [ 0.402243] acpiphp: Slot [3] registered [ 0.403136] acpiphp: Slot [4] registered [ 0.405141] acpiphp: Slot [11] registered [ 0.406122] acpiphp: Slot [12] registered [ 0.407141] acpiphp: Slot [13] registered [ 0.409197] acpiphp: Slot [14] registered [ 0.410150] acpiphp: Slot [15] registered [ 0.412127] acpiphp: Slot [16] registered [ 0.414131] acpiphp: Slot [17] registered [ 0.415147] acpiphp: Slot [18] registered [ 0.417122] acpiphp: Slot [19] registered [ 0.419160] acpiphp: Slot [20] registered [ 0.421151] acpiphp: Slot [21] registered [ 0.424194] acpiphp: Slot [22] registered [ 0.427171] acpiphp: Slot [23] registered [ 0.428127] acpiphp: Slot [24] registered [ 0.430138] acpiphp: Slot [25] registered [ 0.432152] acpiphp: Slot [26] registered [ 0.433299] acpiphp: Slot [27] registered [ 0.435130] acpiphp: Slot [28] registered [ 0.437146] acpiphp: Slot [29] registered [ 0.439147] acpiphp: Slot [30] registered [ 0.441181] acpiphp: Slot [31] registered [ 0.442101] PCI host bridge to bus 0000:00 [ 0.444038] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.447042] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.449040] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.452054] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.455038] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.457036] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.460308] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.463236] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.467345] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.480015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.485020] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.487024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.490023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.494023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.496564] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.499779] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.501047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.504733] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.510029] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.523956] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.530024] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.536903] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.546021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.552017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.568029] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.581121] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.588023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.598026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.614020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.627915] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.634029] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.643025] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.659022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.674545] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.679026] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.685030] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.713025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.725236] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.732016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.741023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.761016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.770734] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.783024] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.797025] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.819024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.833064] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.835454] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.838384] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.841401] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.843254] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.849005] iommu: Default domain type: Passthrough [ 0.851479] SCSI subsystem initialized [ 0.853173] ACPI: bus type USB registered [ 0.854121] usbcore: registered new interface driver usbfs [ 0.856101] usbcore: registered new interface driver hub [ 0.858099] usbcore: registered new device driver usb [ 0.860212] pps_core: LinuxPPS API ver. 1 registered [ 0.862014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.865063] PTP clock support registered [ 0.867166] EDAC MC: Ver: 3.0.0 [ 0.869108] PCI: Using ACPI for IRQ routing [ 0.870642] NetLabel: Initializing [ 0.872017] NetLabel: domain hash size = 128 [ 0.874012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.875073] NetLabel: unlabeled traffic allowed by default [ 0.877125] vgaarb: loaded [ 0.878302] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.881016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.886393] clocksource: Switched to clocksource kvm-clock [ 0.995539] VFS: Disk quotas dquot_6.6.0 [ 0.997180] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.999846] *** VALIDATE ramfs *** [ 1.001232] *** VALIDATE hugetlbfs *** [ 1.002587] pnp: PnP ACPI init [ 1.004861] pnp: PnP ACPI: found 6 devices [ 1.029043] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.032864] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.035304] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.037917] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.040686] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.043493] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.046367] NET: Registered protocol family 2 [ 1.048961] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.054590] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.058506] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.064653] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.067715] TCP: Hash tables configured (established 65536 bind 65536) [ 1.070833] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.073751] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.076897] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.079993] NET: Registered protocol family 1 [ 1.083824] RPC: Registered named UNIX socket transport module. [ 1.086170] RPC: Registered udp transport module. [ 1.087691] RPC: Registered tcp transport module. [ 1.089471] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.091759] NET: Registered protocol family 44 [ 1.093399] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.095784] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.098268] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.100660] PCI: CLS 0 bytes, default 64 [ 1.102414] Unpacking initramfs... [ 2.570322] debug: unmapping init [mem 0xffff8c533cc54000-0xffff8c533ffbffff] [ 2.574782] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.578062] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.581441] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 3.117211] Initialise system trusted keyrings [ 3.119420] Key type blacklist registered [ 3.122030] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.133394] zbud: loaded [ 3.137195] *** VALIDATE nfs *** [ 3.138184] *** VALIDATE nfs4 *** [ 3.139511] pstore: using deflate compression [ 3.142324] Platform Keyring initialized [ 3.256581] NET: Registered protocol family 38 [ 3.258454] Key type asymmetric registered [ 3.259528] Asymmetric key parser 'x509' registered [ 3.261735] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.265104] io scheduler mq-deadline registered [ 3.266656] io scheduler kyber registered [ 3.268223] io scheduler bfq registered [ 3.270594] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.274282] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.277836] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.280876] ACPI: Power Button [PWRF] [ 3.288458] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.296376] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.312162] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.320769] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.334738] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.364532] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.393488] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.398547] Non-volatile memory driver v1.3 [ 3.400559] Linux agpgart interface v0.103 [ 3.434065] virtio_blk virtio1: [vda] 145920 512-byte logical blocks (74.7 MB/71.3 MiB) [ 3.438504] vda: detected capacity change from 0 to 74711040 [ 3.452738] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.455399] vdb: detected capacity change from 0 to 1073741824 [ 3.469552] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.473284] vdc: detected capacity change from 0 to 2621440000 [ 3.492303] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.495545] vdd: detected capacity change from 0 to 2621440000 [ 3.509952] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.512625] vde: detected capacity change from 0 to 4294967296 [ 3.527575] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.531454] vdf: detected capacity change from 0 to 4294967296 [ 3.543976] libphy: Fixed MDIO Bus: probed [ 3.552437] usbcore: registered new interface driver usbserial_generic [ 3.555373] usbserial: USB Serial support registered for generic [ 3.557680] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.562197] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.564503] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.567530] mousedev: PS/2 mouse device common for all mice [ 3.571050] rtc_cmos 00:05: RTC can wake from S4 [ 3.573933] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.578818] rtc_cmos 00:05: registered as rtc0 [ 3.582101] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.583345] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.586379] intel_pstate: CPU model not supported [ 3.593597] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.597812] hid: raw HID events driver (C) Jiri Kosina [ 3.602822] usbcore: registered new interface driver usbhid [ 3.606165] usbhid: USB HID core driver [ 3.608411] drop_monitor: Initializing network drop monitor service [ 3.611983] Initializing XFRM netlink socket [ 3.615059] NET: Registered protocol family 10 [ 3.619285] Segment Routing with IPv6 [ 3.621119] NET: Registered protocol family 17 [ 3.623745] mpls_gso: MPLS GSO support [ 3.632413] RAS: Correctable Errors collector initialized. [ 3.634980] AVX version of gcm_enc/dec engaged. [ 3.636750] AES CTR mode by8 optimization enabled [ 3.743481] sched_clock: Marking stable (3743449177, 0)->(4678701045, -935251868) [ 3.747453] registered taskstats version 1 [ 3.749764] Loading compiled-in X.509 certificates [ 3.753204] zswap: loaded using pool lzo/zbud [ 3.780072] Key type big_key registered [ 3.793864] Key type encrypted registered [ 3.795294] ima: No TPM chip found, activating TPM-bypass! [ 3.797634] ima: Allocated hash algorithm: sha1 [ 3.799504] ima: No architecture policies found [ 3.801631] evm: Initialising EVM extended attributes: [ 3.804115] evm: security.selinux [ 3.805578] evm: security.ima [ 3.806873] evm: security.capability [ 3.808247] evm: HMAC attrs: 0x1 [ 3.810884] rtc_cmos 00:05: setting system clock to 2026-08-19 06:02:44 UTC (1787119364) [ 3.818721] debug: unmapping init [mem 0xffffffff9d803000-0xffffffff9d9fffff] [ 3.821970] debug: unmapping init [mem 0xffffffff9c582000-0xffffffff9c858fff] [ 3.834378] Write protecting the kernel read-only data: 28672k [ 3.838037] debug: unmapping init [mem 0xffffffff9ac03000-0xffffffff9adfffff] [ 3.841377] debug: unmapping init [mem 0xffffffff9b514000-0xffffffff9b5fffff] [ 3.880122] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.889777] systemd[1]: Detected virtualization kvm. [ 3.891740] systemd[1]: Detected architecture x86-64. [ 3.893613] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.917796] systemd[1]: No hostname configured. [ 3.919755] systemd[1]: Set hostname to . [ 3.921539] random: systemd: uninitialized urandom read (16 bytes read) [ 3.923551] systemd[1]: Initializing machine ID from random generator. [ 3.961519] random: ln: uninitialized urandom read (6 bytes read) [ 4.071576] random: systemd: uninitialized urandom read (16 bytes read) [ 4.074544] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.078872] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.083063] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.752947] device-mapper: uevent: version 1.0.3 [ 4.755649] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK [ 5.010391] random: fast init done ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.520763] virtio_net virtio0 ens2: renamed from eth0 [ 5.574281] scsi host0: ata_piix [ 5.613832] scsi host1: ata_piix [ 5.616344] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.618851] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.427968] dracut-initqueue[577]: RTNETLINK answers: File exists [ 10.313408] random: crng init done [ 10.315879] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.892308] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.148586] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.412131] SELinux: Disabled at runtime. [ 12.475039] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.484593] systemd[1]: Detected virtualization kvm. [ 12.486643] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.060722] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.064067] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.069258] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.072692] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.075382] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.085917] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.094681] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ 13.203935] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 13.629394] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.984523] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.039868] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.163255] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.174305] EDAC sbridge: Ver: 1.1.2 [ 16.065739] Key type dns_resolver registered [ 16.376895] NFS: Registering the id_resolver key type [ 16.379328] Key type id_resolver registered [ 16.380977] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg442-server login: [ 63.098133] hrtimer: interrupt took 4124295 ns [ 93.481705] libcfs: loading out-of-tree module taints kernel. [ 93.532374] Key type ._llcrypt registered [ 93.534840] Key type .llcrypt registered [ 93.633409] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 113.923083] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 115.345341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 115.359789] alg: No test for adler32 (adler32-zlib) [ 117.359528] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 118.512274] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 120.328205] Key type lgssc registered [ 122.795856] Lustre: Echo OBD driver; http://www.lustre.org/ [ 143.661770] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 186.783262] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 200.115488] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 200.139403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 201.382132] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 201.419578] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 201.514585] Lustre: lustre-MDT0000: new disk, initializing [ 201.588840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 201.609065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 205.674668] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 218.507422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 218.669382] Lustre: 6511:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 218.713061] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 218.716352] Lustre: Skipped 1 previous similar message [ 218.840582] Lustre: lustre-MDT0001: new disk, initializing [ 218.947168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 218.996335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 219.013908] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 223.604566] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 228.581765] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 238.046822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 238.280590] Lustre: lustre-OST0000: new disk, initializing [ 238.284713] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 238.289828] Lustre: 8448:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 238.364704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 243.845701] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 244.808450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 244.814149] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 244.914180] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 259.076168] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 259.324704] Lustre: lustre-OST0001: new disk, initializing [ 259.335250] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 259.346398] Lustre: 9521:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 259.430105] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 265.282345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 265.301567] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 265.367683] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 266.451288] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 278.316865] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 287.951206] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 294.122271] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing check_logdir /tmp/testlogs/ [ 298.739299] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing yml_node [ 303.276048] Lustre: DEBUG MARKER: Client: 2.17.57.3 [ 306.094262] Lustre: DEBUG MARKER: MDS: 2.17.57.3 [ 308.444831] Lustre: DEBUG MARKER: OSS: 2.17.57.3 [ 310.138677] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 19 02:07:49 EDT 2026 [ 325.320967] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 326.795793] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 347.105792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 347.107046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 347.124547] Lustre: Skipped 3 previous similar messages [ 347.142661] Lustre: Skipped 3 previous similar messages [ 349.185387] Lustre: server umount lustre-MDT0000 complete [ 353.221223] LustreError: 9520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787119713 with bad export cookie 6764225971307450951 [ 353.223879] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 353.227044] LustreError: 9520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 357.346953] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 357.349109] LustreError: 10098:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 357.356247] Lustre: Skipped 1 previous similar message [ 357.357556] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 357.365464] LustreError: 10098:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 358.371213] LustreError: 6518:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 358.371737] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 358.384566] Lustre: Skipped 2 previous similar messages [ 359.540181] Lustre: server umount lustre-MDT0001 complete [ 362.668526] Lustre: server umount lustre-OST0000 complete [ 365.793858] Lustre: server umount lustre-OST0001 complete [ 372.075167] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 380.580588] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 413.451202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 413.782835] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 413.827696] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 413.961159] Lustre: lustre-MDT0000: new disk, initializing [ 414.031631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 414.054308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 417.988943] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 428.421749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 428.630103] Lustre: 15074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 428.738248] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 428.744513] Lustre: Skipped 1 previous similar message [ 428.860157] Lustre: lustre-MDT0001: new disk, initializing [ 429.041331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 429.092797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 429.107171] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 433.470634] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 442.860478] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 447.619405] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 454.623304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 454.825255] Lustre: lustre-OST0000: new disk, initializing [ 454.828568] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 454.833279] Lustre: 17040:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 454.914739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 456.448656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 456.458807] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 456.580920] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 461.467789] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 471.003882] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 476.051333] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 476.243105] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 480.784748] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 480.956280] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 487.395601] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 487.398623] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 487.400952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 487.402690] Lustre: Skipped 1 previous similar message [ 492.532438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 492.542156] Lustre: Skipped 2 previous similar messages [ 497.120181] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 497.323745] Lustre: server umount lustre-OST0000 complete [ 502.754232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 502.757319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 502.779283] Lustre: Skipped 1 previous similar message [ 507.205980] Lustre: server umount lustre-MDT0000 complete [ 507.874776] LustreError: 15079:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 507.903819] LustreError: 15079:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 510.810822] LustreError: 15064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787119871 with bad export cookie 6764225971307452484 [ 510.840499] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 511.133662] Lustre: server umount lustre-MDT0001 complete [ 516.055265] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 02:11:15 (1787119875) [ 522.634215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 522.958548] LustreError: 19305:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 523.017856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 527.055699] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 528.366318] LustreError: 19306:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 528.391096] LustreError: 19306:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 534.870062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 539.077819] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 546.320126] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 551.434531] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 557.945759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 564.281337] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 572.456448] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 575.965729] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 583.140887] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 583.143892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 583.152318] Lustre: Skipped 2 previous similar messages [ 583.165723] Lustre: Skipped 3 previous similar messages [ 588.707062] Lustre: server umount lustre-MDT0000 complete [ 591.975665] LustreError: 19287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787119952 with bad export cookie 6764225971307453366 [ 591.980623] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 591.989663] LustreError: 19287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 592.410059] Lustre: server umount lustre-MDT0001 complete [ 605.581284] Lustre: server umount lustre-OST0000 complete [ 626.240313] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 635.930556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 636.176410] LustreError: 23682:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 636.199868] LustreError: 23682:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 636.260136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 636.267636] Lustre: Skipped 2 previous similar messages [ 640.265707] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 648.476173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 654.169139] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 656.915246] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 663.397097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 669.316592] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 677.072072] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 677.256707] Lustre: lustre-OST0001: new disk, initializing [ 677.262403] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 677.269745] Lustre: 26000:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 679.001140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 679.012276] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 679.099453] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 682.668551] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 691.737592] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 695.259204] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 710.636942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 710.649394] LustreError: Skipped 1 previous similar message [ 710.654136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 710.670191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 710.674484] Lustre: Skipped 3 previous similar messages [ 714.351604] Lustre: server umount lustre-MDT0000 complete [ 717.282991] LustreError: 23683:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 717.319294] LustreError: 23683:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 718.553819] LustreError: 25999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120079 with bad export cookie 6764225971307454213 [ 718.566455] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 718.584610] LustreError: 25999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 719.085096] Lustre: server umount lustre-MDT0001 complete [ 733.109380] Lustre: server umount lustre-OST0000 complete [ 747.010502] Lustre: server umount lustre-OST0001 complete [ 763.394817] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 772.974854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 773.397070] LustreError: 29020:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 773.485628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 773.492054] Lustre: Skipped 3 previous similar messages [ 777.940543] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 785.748106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 789.958331] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 792.540163] Lustre: 30170:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 798.119405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 803.969446] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 812.106723] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 818.808650] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 827.602315] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 831.915811] Lustre: 32047:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 838.114868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 838.127292] Lustre: Skipped 5 previous similar messages [ 838.136948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 838.144973] Lustre: Skipped 6 previous similar messages [ 844.506867] Lustre: server umount lustre-MDT0000 complete [ 848.367147] LustreError: 29757:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 848.395836] LustreError: 29757:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 848.438639] LustreError: 29001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120209 with bad export cookie 6764225971307455508 [ 848.442583] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 848.447706] LustreError: 29001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 848.847242] Lustre: server umount lustre-MDT0001 complete [ 862.806416] Lustre: server umount lustre-OST0000 complete [ 876.186238] Lustre: server umount lustre-OST0001 complete [ 884.717331] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 02:17:24 (1787120244) [ 910.508521] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 919.737513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 924.917550] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 932.643960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 937.658254] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 940.591349] Lustre: 35998:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 946.591410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 952.981394] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 962.625087] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 969.473883] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 976.782627] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 980.057745] Lustre: 37879:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 985.574816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 985.585518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 985.591267] Lustre: Skipped 2 previous similar messages [ 985.599133] Lustre: Skipped 5 previous similar messages [ 990.702300] LustreError: 34845:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 990.730166] LustreError: 34845:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 990.752567] Lustre: server umount lustre-MDT0000 complete [ 993.781715] LustreError: 34830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120354 with bad export cookie 6764225971307456614 [ 993.791911] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 993.792752] LustreError: 34830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 994.101469] Lustre: server umount lustre-MDT0001 complete [ 1006.501780] Lustre: server umount lustre-OST0000 complete [ 1019.683393] Lustre: server umount lustre-OST0001 complete [ 1026.362590] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 02:19:46 (1787120386) [ 1040.289334] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 1049.444415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1049.925105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1049.935550] Lustre: Skipped 7 previous similar messages [ 1053.785792] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1060.876545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1064.435398] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1066.784371] Lustre: 41169:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1072.133038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1078.937256] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1087.544673] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1093.745541] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1100.940352] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1104.888464] Lustre: 43045:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1105.899808] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1105.913453] Lustre: Skipped 1 previous similar message [ 1112.546901] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1112.559833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1112.569701] Lustre: Skipped 1 previous similar message [ 1115.036735] Lustre: server umount lustre-MDT0000 complete [ 1118.278589] LustreError: 40001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120478 with bad export cookie 6764225971307457720 [ 1118.280923] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1118.303066] LustreError: 40001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1124.777552] Lustre: server umount lustre-MDT0001 complete [ 1133.459534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1143.265332] LustreError: 3640:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8c538353ed80 x1873930595207424/t0(0) o250->MGC192.168.204.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1147.873274] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1152.949123] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1156.193081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1160.894148] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1161.728546] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1161.739962] Lustre: Skipped 1 previous similar message [ 1167.715977] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1169.480424] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1174.962349] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1176.605717] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1182.179931] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1187.494189] Lustre: server umount lustre-MDT0000 complete [ 1191.049910] LustreError: 40001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120551 with bad export cookie 6764225971307458812 [ 1191.052766] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1191.058352] LustreError: 40001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1191.422501] Lustre: server umount lustre-MDT0001 complete [ 1205.295211] Lustre: server umount lustre-OST0000 complete [ 1209.313512] Lustre: 3641:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787120553/real 1787120553] req@ffff8c53b5226a00 x1873930595240448/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787120569 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1209.905715] Lustre: server umount lustre-OST0001 complete [ 1220.343663] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 02:22:59 (1787120579) [ 1237.086756] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 1247.345154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1247.799865] LustreError: 47653:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1247.814805] LustreError: 47653:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 20 previous similar messages [ 1252.526691] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1262.018127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1266.899377] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1269.835531] Lustre: 48805:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1276.779310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1284.240104] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1292.556603] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1299.041476] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1306.961377] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1311.684202] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1324.001786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1324.005888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1324.025556] Lustre: Skipped 10 previous similar messages [ 1324.033399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1324.039476] Lustre: Skipped 20 previous similar messages [ 1338.336316] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1338.509475] Lustre: server umount lustre-MDT0000 complete [ 1342.122291] LustreError: 47633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120702 with bad export cookie 6764225971307459638 [ 1342.123355] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1342.137678] LustreError: 47633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1342.696713] Lustre: server umount lustre-MDT0001 complete [ 1356.683729] Lustre: server umount lustre-OST0000 complete [ 1359.328110] Lustre: 3642:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787120704/real 1787120704] req@ffff8c53b7f71500 x1873930595298304/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787120720 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1361.131745] Lustre: server umount lustre-OST0001 complete [ 1371.624819] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 02:25:31 (1787120731) [ 1373.883802] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1375.806822] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 02:25:35 (1787120735) [ 1388.711196] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 1397.877536] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 1419.773715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1419.979647] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1420.091724] Lustre: lustre-MDT0000: new disk, initializing [ 1420.181186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1425.462447] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1435.945995] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1442.779575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1442.970237] Lustre: lustre-OST0000: new disk, initializing [ 1442.976491] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1442.983343] Lustre: Skipped 1 previous similar message [ 1442.988605] Lustre: 54888:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1444.904656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1444.917199] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1444.960878] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1448.705301] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1457.154392] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1459.235519] Lustre: DEBUG MARKER: create test files [ 1502.109531] Lustre: server umount lustre-OST0000 complete [ 1506.182416] Lustre: server umount lustre-MDT0000 complete [ 1516.698174] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 1519.703779] Key type lgssc unregistered [ 1519.997179] LNet: 56522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1520.004881] LNetError: 56522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1520.042730] LNet: Removed LNI 192.168.204.142@tcp [ 1521.276203] Key type .llcrypt unregistered [ 1521.279978] Key type ._llcrypt unregistered [ 1547.272601] Key type ._llcrypt registered [ 1547.277393] Key type .llcrypt registered [ 1547.393149] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 1548.583387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1548.611050] alg: No test for adler32 (adler32-zlib) [ 1549.685543] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 1550.016992] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 1551.713246] Key type lgssc registered [ 1552.788786] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1560.896692] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1560.917637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1562.365787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1566.507854] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1573.392817] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1580.088901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1580.370149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1586.777210] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1587.587054] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1587.750384] Lustre: 59247:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 510 < left 544, rollback = 9 [ 1587.761942] Lustre: 59247:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/5, destroy: 0/0/0 [ 1587.768235] Lustre: 59247:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1587.776615] Lustre: 59247:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1587.781363] Lustre: 59247:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1587.786726] Lustre: 59247:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1594.652986] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1599.880083] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1605.702454] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1613.281445] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1613.296735] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1613.328419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1618.404145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1619.225929] Lustre: server umount lustre-OST0000 complete [ 1623.282614] Lustre: server umount lustre-MDT0000 complete [ 1636.197960] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 1639.026916] Key type lgssc unregistered [ 1639.389740] LNet: 60857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1639.399679] LNetError: 60857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1640.440045] LNet: Removed LNI 192.168.204.142@tcp [ 1641.334388] Key type .llcrypt unregistered [ 1641.336558] Key type ._llcrypt unregistered [ 1675.029634] Key type ._llcrypt registered [ 1675.032788] Key type .llcrypt registered [ 1675.315704] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 1677.823792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1677.852465] alg: No test for adler32 (adler32-zlib) [ 1678.979166] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 1679.202610] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 1680.904254] Key type lgssc registered [ 1682.038726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1690.267323] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1690.320848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1691.717243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1695.972153] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1706.369221] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1716.268880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1716.913974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1717.946066] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1718.085642] Lustre: 63699:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1718.096869] Lustre: 63699:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1718.118531] Lustre: 63699:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1718.132626] Lustre: 63699:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1718.154143] Lustre: 63699:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1718.168098] Lustre: 63699:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1725.687820] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1733.478886] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1738.716679] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1748.963208] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1748.983624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1753.275622] Lustre: server umount lustre-OST0000 complete [ 1756.989533] Lustre: server umount lustre-MDT0000 complete [ 1771.417183] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 1774.585951] Key type lgssc unregistered [ 1775.011065] LNet: 65258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1775.021306] LNetError: 65258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1775.057718] LNet: Removed LNI 192.168.204.142@tcp [ 1776.094240] Key type .llcrypt unregistered [ 1776.096638] Key type ._llcrypt unregistered [ 1797.877228] Key type ._llcrypt registered [ 1797.879026] Key type .llcrypt registered [ 1798.057927] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 1816.913440] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 1818.158463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1818.194399] alg: No test for adler32 (adler32-zlib) [ 1819.311729] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 1819.635399] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 1821.344200] Key type lgssc registered [ 1822.721802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1865.265122] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1865.303619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1866.583806] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1866.614907] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1866.731394] Lustre: lustre-MDT0000: new disk, initializing [ 1866.896391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1866.919642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1872.119804] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1883.652297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1883.801845] Lustre: 68412:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1883.894849] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1883.904602] Lustre: Skipped 1 previous similar message [ 1884.030400] Lustre: lustre-MDT0001: new disk, initializing [ 1884.161243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1884.201947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1884.212909] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1889.302314] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1898.937738] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1905.110498] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1912.994667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1913.373375] Lustre: lustre-OST0000: new disk, initializing [ 1913.380906] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1913.390414] Lustre: 70385:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1913.559128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1919.570061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1919.580765] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1919.727558] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1919.829128] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1929.314244] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1934.129146] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1934.328526] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1938.803433] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1939.059682] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1945.585457] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1945.600519] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1945.608290] LustreError: Skipped 1 previous similar message [ 1945.631786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1945.641923] Lustre: Skipped 1 previous similar message [ 1950.693687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1950.711515] Lustre: Skipped 1 previous similar message [ 1955.296557] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1955.441246] Lustre: server umount lustre-OST0000 complete [ 1960.933482] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1960.934837] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1960.951226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1960.972687] Lustre: Skipped 2 previous similar messages [ 1964.920033] Lustre: server umount lustre-MDT0000 complete [ 1969.447573] LustreError: 69209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121330 with bad export cookie 9807699818620225228 [ 1969.453770] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1969.463902] LustreError: 69209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1969.952149] Lustre: server umount lustre-MDT0001 complete [ 1978.596279] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 02:35:38 (1787121338) [ 1987.907489] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2028.116196] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 2038.917273] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2075.228301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2075.434519] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2075.475606] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2075.532234] Lustre: lustre-MDT0000: new disk, initializing [ 2075.592605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2075.610602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2080.591722] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2093.507012] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2093.683521] Lustre: 76414:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2093.833729] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2093.845908] Lustre: Skipped 1 previous similar message [ 2093.942511] Lustre: lustre-MDT0001: new disk, initializing [ 2094.051925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2094.106146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2094.121131] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2099.580725] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2109.381263] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2115.202379] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2122.009928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2122.426951] Lustre: lustre-OST0000: new disk, initializing [ 2122.430951] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2122.439573] Lustre: 78382:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2122.597411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2128.673781] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2131.017881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2131.028405] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2131.132585] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2137.651873] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2141.469742] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2141.647600] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2146.049819] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2146.256630] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2151.907305] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2151.912995] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2151.916209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2151.928436] Lustre: Skipped 1 previous similar message [ 2151.946416] Lustre: Skipped 2 previous similar messages [ 2154.369646] Lustre: server umount lustre-OST0000 complete [ 2160.612964] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2160.624786] LustreError: Skipped 1 previous similar message [ 2160.631517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2160.655948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2164.531740] Lustre: server umount lustre-MDT0000 complete [ 2167.267456] LustreError: 79842:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2167.293207] LustreError: 79842:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2167.943445] LustreError: 76405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121528 with bad export cookie 9807699818620226110 [ 2167.947627] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2167.953573] LustreError: 76405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2168.264255] Lustre: server umount lustre-MDT0001 complete [ 2177.114704] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 02:38:56 (1787121536) [ 2184.181314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2184.672555] LustreError: 80792:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2184.750432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2189.795509] LustreError: 80791:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2190.086840] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2194.921383] LustreError: 80792:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2199.735056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2200.041750] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2200.045766] Lustre: Skipped 2 previous similar messages [ 2204.534378] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2211.915370] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2217.615239] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2223.893679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2224.341962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2224.346188] Lustre: Skipped 1 previous similar message [ 2230.233216] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2237.616280] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2241.241740] Lustre: 83463:0:(genops.c:1795:obd_export_evict_by_uuid()) MGS: evicting 1c344aba-5354-422f-851d-93a4e04b0b95 at adminstrative request [ 2243.049916] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2243.073348] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2253.293490] Lustre: MGS: Client 1c344aba-5354-422f-851d-93a4e04b0b95 (at 0@lo) reconnecting [ 2253.303626] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x881bf30588858db0 to 0x881bf30588859050 [ 2253.313483] Lustre: MGC192.168.204.142@tcp: Connection restored to 0@lo (at 0@lo) [ 2278.888179] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2279.098719] Lustre: lustre-OST0001: new disk, initializing [ 2279.102226] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2279.107779] Lustre: 83854:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2285.943198] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2286.739341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2286.751447] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2286.871262] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2296.305424] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2302.444271] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2302.445871] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2302.450466] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2302.455910] LustreError: Skipped 1 previous similar message [ 2302.484387] Lustre: Skipped 2 previous similar messages [ 2306.111988] Lustre: server umount lustre-OST0001 complete [ 2307.585574] LustreError: 82610:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2307.625679] LustreError: 82610:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2311.658518] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2311.671037] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2316.884199] Lustre: server umount lustre-OST0000 complete [ 2327.847772] Lustre: server umount lustre-MDT0000 complete [ 2328.035235] LustreError: 80791:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2328.060631] LustreError: 80791:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2331.876887] LustreError: 80772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121692 with bad export cookie 9807699818620227664 [ 2331.882401] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2331.897720] LustreError: 80772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2332.104552] Lustre: server umount lustre-MDT0001 complete [ 2341.643915] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 02:41:41 (1787121701) [ 2343.449593] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2345.237520] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 02:41:44 (1787121704) [ 2346.780697] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2348.589077] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 02:41:48 (1787121708) [ 2350.321666] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2352.332253] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 02:41:51 (1787121711) [ 2353.711293] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2355.919378] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 02:41:55 (1787121715) [ 2357.759931] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2363.460653] loop: module loaded [ 2363.478968] loop0: detected capacity change from 0 to 4294967296 [ 2363.541177] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2365.116924] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2368.994246] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2373.028489] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2397.982181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2398.044202] LustreError: 87978:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2398.049477] LustreError: 87978:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2398.054400] LustreError: 87978:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2398.059414] LustreError: 87978:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2418.240369] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 2428.602583] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2465.973764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2466.135684] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2466.153855] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2466.239864] Lustre: lustre-MDT0000: new disk, initializing [ 2466.337559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2466.345721] Lustre: Skipped 1 previous similar message [ 2466.361225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2470.831871] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2482.048934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2482.209487] Lustre: 90843:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2482.585584] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2487.263857] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2497.104131] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2503.578231] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2510.990342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2511.224960] Lustre: 92811:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2517.588375] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2517.693700] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2518.028687] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2527.459938] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2533.545206] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2533.787598] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2539.638195] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2539.949246] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2543.586091] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2543.586895] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2543.605861] LustreError: Skipped 1 previous similar message [ 2543.609731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2543.609740] Lustre: Skipped 6 previous similar messages [ 2543.687174] Lustre: Skipped 4 previous similar messages [ 2548.756424] LustreError: 92795:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2548.933514] Lustre: server umount lustre-OST0000 complete [ 2560.167592] Lustre: server umount lustre-MDT0000 complete [ 2564.994062] LustreError: 90837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121925 with bad export cookie 9807699818620228280 [ 2565.001099] LustreError: 90837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2565.007660] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2565.472990] Lustre: server umount lustre-MDT0001 complete [ 2576.215402] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 02:45:35 (1787121935) [ 2583.578635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2584.388643] LustreError: 95224:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2584.400930] LustreError: 95224:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2590.410335] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2599.679428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2600.153107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-60 [ 2600.165754] Lustre: Skipped 3 previous similar messages [ 2604.539704] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2612.406893] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2618.541465] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2625.924724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2633.386562] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2641.779227] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2646.719622] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2647.126098] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2651.626272] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2651.861957] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2658.458025] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2658.631439] Lustre: lustre-OST0001: new disk, initializing [ 2658.634654] Lustre: Skipped 2 previous similar messages [ 2658.637881] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2658.641674] Lustre: Skipped 3 previous similar messages [ 2658.645764] Lustre: 98597:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2664.971719] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2668.693516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2668.703489] Lustre: Skipped 2 previous similar messages [ 2668.714831] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2668.832273] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2675.330992] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2680.303613] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2680.587490] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2685.358542] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2685.559439] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2709.576906] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2738.207294] Lustre: Failing over lustre-MDT0000 [ 2739.029186] Lustre: server umount lustre-MDT0000 complete [ 2739.424700] LustreError: 95221:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2739.444088] LustreError: 95221:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2740.709146] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2740.723896] Lustre: Skipped 2 previous similar messages [ 2757.088650] Lustre: 66638:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787122101/real 1787122101] req@00000000d1260693 x1873932378719744/t0(0) o400->MGC192.168.204.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787122117 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2757.121960] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2766.556721] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2766.558921] LDISKFS-fs (dm-0): recovery complete [ 2766.591320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2767.340079] LustreError: 66635:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ea858e72 x1873932378728832/t0(0) o250->MGC192.168.204.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2767.912679] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2770.126725] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2770.147980] Lustre: 95221:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2772.775361] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2772.965237] Lustre: 95224:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2772.975606] Lustre: 95224:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 1 previous similar message [ 2773.000948] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2773.109926] LustreError: 101446:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2773.450485] LustreError: 101446:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2773.792801] LustreError: 101446:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2773.823890] LustreError: 101446:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 1 previous similar message [ 2774.160860] LustreError: 101446:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2774.176983] LustreError: 101446:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 1 previous similar message [ 2774.195244] Lustre: 101446:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2774.209876] Lustre: 101446:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 7 previous similar messages [ 2774.857249] LustreError: 101446:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2774.877245] LustreError: 101446:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 2 previous similar messages [ 2775.201615] LustreError: 101446:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2775.212044] LustreError: 101446:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 2 previous similar messages [ 2776.507921] Lustre: 101446:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2776.514673] Lustre: 101446:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 6 previous similar messages [ 2777.154229] LustreError: 101446:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2777.161107] LustreError: 101446:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 6 previous similar messages [ 2777.489445] LustreError: 101446:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2777.502218] LustreError: 101446:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 6 previous similar messages [ 2780.699377] Lustre: 101446:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2780.725112] Lustre: 101446:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 12 previous similar messages [ 2781.397043] LustreError: 101446:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2781.411175] LustreError: 101446:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 12 previous similar messages [ 2781.728678] LustreError: 101446:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2781.744239] LustreError: 101446:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 12 previous similar messages [ 2788.805111] Lustre: 101446:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2788.821954] Lustre: 101446:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 24 previous similar messages [ 2789.469481] LustreError: 101446:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2789.486685] LustreError: 101446:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 24 previous similar messages [ 2789.816251] LustreError: 101446:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2789.835518] LustreError: 101446:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 24 previous similar messages [ 2805.113657] Lustre: 101446:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2805.131108] Lustre: 101446:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 50 previous similar messages [ 2805.478458] LustreError: 101446:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2805.489982] LustreError: 101446:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 49 previous similar messages [ 2805.824167] LustreError: 101446:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2805.831568] LustreError: 101446:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 49 previous similar messages [ 2830.276877] Lustre: 101446:0:(ldlm_lib.c:2394:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2830.284655] Lustre: 101446:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2830.291339] Lustre: 101446:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client cfa08969-3e6c-488d-a01d-d9ac0d1bbcaf@192.168.204.42@tcp [ 2830.306985] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2830.311863] Lustre: 101446:0:(ldlm_lib.c:1914:abort_req_replay_queue()) @@@ aborted: req@00000000636551c3 x1873932360685568/t0(8589934775) o101->cfa08969-3e6c-488d-a01d-d9ac0d1bbcaf@192.168.204.42@tcp:206/0 lens 592/0 e 0 to 0 dl 1787122201 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2830.355802] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2830.375161] Lustre: lustre-MDT0000: Denying connection for new client cfa08969-3e6c-488d-a01d-d9ac0d1bbcaf (at 192.168.204.42@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 2830.395845] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2830.433827] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2830.448204] Lustre: Skipped 2 previous similar messages [ 2830.456079] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2830.515461] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:79 to 0x280000401:577) [ 2830.517926] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:46 to 0x2c0000401:545) [ 2844.659097] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2844.669843] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2844.676863] Lustre: Skipped 1 previous similar message [ 2844.680930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2844.684524] Lustre: Skipped 7 previous similar messages [ 2849.630371] Lustre: server umount lustre-OST0000 complete [ 2859.863234] Lustre: server umount lustre-OST0001 complete [ 2870.375780] Lustre: server umount lustre-MDT0000 complete [ 2874.337525] LustreError: 95954:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2874.372539] LustreError: 95954:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 46 previous similar messages [ 2874.527192] LustreError: 95206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122235 with bad export cookie 9807699818620279492 [ 2874.542937] LustreError: 95206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2874.557915] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2875.209938] Lustre: server umount lustre-MDT0001 complete [ 2886.936418] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 02:50:46 (1787122246) [ 2897.396780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2897.409808] Lustre: *** cfs_fail_loc=197, val=0*** [ 2912.930681] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2918.048939] LustreError: 103421:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.142@tcp: failed processing log, type 4: rc = -110 [ 2923.187766] Lustre: *** cfs_fail_loc=197, val=0*** [ 2923.189712] Lustre: Skipped 65 previous similar messages [ 2943.779466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2943.789450] Lustre: Skipped 3 previous similar messages [ 2953.884560] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2962.505649] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2964.704874] Lustre: server umount lustre-OST0000 complete [ 2972.897812] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 02:52:12 (1787122332) [ 2994.405326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2999.490350] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3008.848198] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3013.682921] LustreError: 105075:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3013.700296] LustreError: 105075:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3013.709079] LustreError: 105075:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 3014.281445] Lustre: server umount lustre-MDT0000 complete [ 3052.694546] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 3064.682128] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 3101.005728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3101.309648] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3101.345492] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3101.429784] Lustre: lustre-MDT0000: new disk, initializing [ 3101.528587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3106.416497] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3117.264133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3117.400866] Lustre: 109250:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3117.877975] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3123.238695] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3135.411833] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3143.818068] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3151.811829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3152.312117] Lustre: 111218:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3159.143430] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3162.352889] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3171.139405] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3176.835395] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3177.000108] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3181.603190] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3181.876620] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3188.211703] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3188.223519] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3188.239040] Lustre: Skipped 6 previous similar messages [ 3188.256453] LustreError: Skipped 1 previous similar message [ 3188.266802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3188.280194] Lustre: Skipped 7 previous similar messages [ 3190.459341] Lustre: server umount lustre-OST0000 complete [ 3200.527724] Lustre: server umount lustre-MDT0000 complete [ 3203.558547] LustreError: 109261:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3203.601465] LustreError: 109261:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 3205.190166] LustreError: 110045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122565 with bad export cookie 9807699818620290111 [ 3205.205415] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3205.209025] LustreError: 110045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3205.811937] Lustre: server umount lustre-MDT0001 complete [ 3215.829922] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 02:56:15 (1787122575) [ 3227.329371] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 3231.275616] Key type lgssc unregistered [ 3231.728959] LNet: 113709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3231.751954] LNetError: 113709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3231.781698] LNet: Removed LNI 192.168.204.142@tcp [ 3233.118650] Key type .llcrypt unregistered [ 3233.128954] Key type ._llcrypt unregistered [ 3255.902911] Key type ._llcrypt registered [ 3255.906926] Key type .llcrypt registered [ 3256.039253] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 3273.369603] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 3274.604736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3274.624309] alg: No test for adler32 (adler32-zlib) [ 3275.821323] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 3276.126219] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 3277.841537] Key type lgssc registered [ 3279.025401] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3325.174429] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3325.197533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3326.520612] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3326.567877] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3326.656824] Lustre: lustre-MDT0000: new disk, initializing [ 3326.807358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3326.826441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3331.801699] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3343.294270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3343.734485] Lustre: lustre-OST0000: new disk, initializing [ 3343.745322] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3343.753683] Lustre: Skipped 1 previous similar message [ 3343.761611] Lustre: 117882:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3343.841281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3351.779072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3351.791471] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3351.823558] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3352.079932] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3365.218515] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3365.497554] Lustre: lustre-OST0001: new disk, initializing [ 3365.503802] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3365.514645] Lustre: 118745:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3365.596959] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3371.864169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3371.893864] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3371.940269] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3375.772812] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3387.362300] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3387.367813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3387.386638] Lustre: Skipped 1 previous similar message [ 3387.409551] Lustre: Skipped 1 previous similar message [ 3392.283634] Lustre: server umount lustre-MDT0000 complete [ 3400.091554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3406.171522] LustreError: 117069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122766 with bad export cookie 8064221161858436804 [ 3406.194745] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3406.477241] Lustre: server umount lustre-OST0000 complete [ 3412.247722] Lustre: server umount lustre-OST0001 complete [ 3425.476909] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 02:59:44 (1787122784) [ 3446.836687] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 03:00:06 (1787122806) [ 3456.846564] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 3482.029856] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 03:00:41 (1787122841) [ 3528.246125] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 3542.114270] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 3581.393055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3581.623598] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3581.655729] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3581.757578] Lustre: lustre-MDT0000: new disk, initializing [ 3581.850960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3581.870259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3586.002863] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3596.813655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3596.970506] Lustre: 126515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3597.069684] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3597.080229] Lustre: Skipped 1 previous similar message [ 3597.153513] Lustre: lustre-MDT0001: new disk, initializing [ 3597.250792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3597.285884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3597.299083] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3601.993949] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3611.604117] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3617.271058] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3624.250662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3624.701660] Lustre: lustre-OST0000: new disk, initializing [ 3624.707129] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3624.718466] Lustre: 128490:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3624.886118] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3630.639051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3630.659178] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3630.756908] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3631.137765] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3641.590322] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3646.415522] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3646.659765] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3651.499983] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3651.774853] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3656.677378] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3656.685604] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3656.701526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3656.726658] Lustre: Skipped 1 previous similar message [ 3656.739677] Lustre: Skipped 1 previous similar message [ 3660.296484] Lustre: server umount lustre-OST0000 complete [ 3666.915237] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3666.920427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3666.935043] Lustre: Skipped 1 previous similar message [ 3666.950341] Lustre: Skipped 1 previous similar message [ 3672.210947] Lustre: server umount lustre-MDT0000 complete [ 3676.714769] LustreError: 126509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787123037 with bad export cookie 8064221161858438029 [ 3676.730221] LustreError: 126509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3676.734188] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3677.104203] Lustre: server umount lustre-MDT0001 complete [ 3687.813646] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 03:04:06 (1787123046) [ 3694.590323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3695.210231] LustreError: 130901:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3695.224941] LustreError: 130901:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3695.291886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3700.357681] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3700.706823] LustreError: 130902:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3704.809227] LustreError: 130901:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3709.119523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3714.003522] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3723.913736] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3730.363392] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3737.050859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3737.520555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3737.537636] Lustre: Skipped 1 previous similar message [ 3744.628904] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3752.612816] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3759.622446] Lustre: *** cfs_fail_loc=159, val=0*** [ 3775.234187] Lustre: lustre-MDT0000: Client 7044c0a9-4c25-4db9-b114-5feaf3b46355 (at 192.168.204.42@tcp) reconnecting [ 3776.972877] Lustre: *** cfs_fail_loc=159, val=0*** [ 3776.977986] Lustre: Skipped 5 previous similar messages [ 3793.164211] Lustre: lustre-MDT0000: Client 7044c0a9-4c25-4db9-b114-5feaf3b46355 (at 192.168.204.42@tcp) reconnecting [ 3811.299760] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3811.307324] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3811.313242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3811.317318] Lustre: Skipped 1 previous similar message [ 3811.345670] LustreError: Skipped 1 previous similar message [ 3816.273029] Lustre: server umount lustre-OST0000 complete [ 3821.539189] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3821.544437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3821.552092] Lustre: Skipped 1 previous similar message [ 3821.558448] Lustre: Skipped 1 previous similar message [ 3831.778636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3831.786316] Lustre: Skipped 4 previous similar messages [ 3834.336153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3834.487973] Lustre: server umount lustre-MDT0000 complete [ 3836.898211] LustreError: 130902:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3836.916152] LustreError: 130902:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3837.981470] LustreError: 133629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787123198 with bad export cookie 8064221161858438911 [ 3837.986026] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3837.999063] LustreError: 133629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3838.295624] Lustre: server umount lustre-MDT0001 complete [ 3851.325400] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 3854.621878] Key type lgssc unregistered [ 3854.902365] LNet: 134690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3854.912077] LNetError: 134690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3854.951270] LNet: Removed LNI 192.168.204.142@tcp [ 3856.069664] Key type .llcrypt unregistered [ 3856.072844] Key type ._llcrypt unregistered [ 3883.739751] Key type ._llcrypt registered [ 3883.741829] Key type .llcrypt registered [ 3883.825980] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 3884.952693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3884.991450] alg: No test for adler32 (adler32-zlib) [ 3886.038472] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 3886.366981] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 3888.064449] Key type lgssc registered [ 3889.253660] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3897.140221] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3897.160055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3898.658889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3903.279741] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3909.091384] LustreError: 136105:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3909.103819] LustreError: 136105:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3911.375043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3911.786986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3916.624514] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3924.990698] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3931.026895] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3937.422532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3937.678223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3943.867163] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3944.192612] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3951.395741] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3964.423741] Lustre: *** cfs_fail_loc=159, val=0*** [ 3980.077037] Lustre: lustre-MDT0000: Client 867c66f1-2238-4546-b710-3cd1cce69986 (at 192.168.204.42@tcp) reconnecting [ 3982.281304] Lustre: *** cfs_fail_loc=159, val=0*** [ 3982.298494] Lustre: Skipped 1 previous similar message [ 3997.999842] Lustre: lustre-MDT0001: Client 867c66f1-2238-4546-b710-3cd1cce69986 (at 192.168.204.42@tcp) reconnecting [ 4000.012499] Lustre: *** cfs_fail_loc=159, val=0*** [ 4000.017451] Lustre: Skipped 2 previous similar messages [ 4016.391272] Lustre: lustre-MDT0001: Client 867c66f1-2238-4546-b710-3cd1cce69986 (at 192.168.204.42@tcp) reconnecting [ 4022.442901] Lustre: 139139:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4025.319908] Lustre: *** cfs_fail_loc=159, val=0*** [ 4025.321519] Lustre: Skipped 4 previous similar messages [ 4041.000413] Lustre: lustre-MDT0000: Client b8b6565d-132a-44d8-886e-951745afed6b (at 192.168.204.42@tcp) reconnecting [ 4042.850129] Lustre: *** cfs_fail_loc=159, val=0*** [ 4042.860178] Lustre: Skipped 14 previous similar messages [ 4058.429770] Lustre: lustre-MDT0000: Client b8b6565d-132a-44d8-886e-951745afed6b (at 192.168.204.42@tcp) reconnecting [ 4060.589888] Lustre: 139431:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4080.099546] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4080.122818] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4080.128174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4085.229954] LustreError: 139445:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4085.503398] Lustre: server umount lustre-OST0000 complete [ 4090.850411] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4090.862118] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4090.872076] Lustre: Skipped 1 previous similar message [ 4090.879393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4090.887691] Lustre: Skipped 1 previous similar message [ 4095.460059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4095.463948] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4097.156214] Lustre: server umount lustre-MDT0000 complete [ 4100.578510] LustreError: 139037:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4100.606950] LustreError: 139037:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4101.304544] LustreError: 138918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787123461 with bad export cookie 7163543111465051616 [ 4101.306221] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4101.313531] LustreError: 138918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4101.668204] Lustre: server umount lustre-MDT0001 complete [ 4113.197900] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 4116.304152] Key type lgssc unregistered [ 4116.657537] LNet: 140514:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4116.667899] LNetError: 140514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4116.695976] LNet: Removed LNI 192.168.204.142@tcp [ 4117.847906] Key type .llcrypt unregistered [ 4117.850810] Key type ._llcrypt unregistered [ 4143.598068] Key type ._llcrypt registered [ 4143.602852] Key type .llcrypt registered [ 4143.776858] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 4144.862556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4144.933085] alg: No test for adler32 (adler32-zlib) [ 4146.109331] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 4146.386423] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 4148.096396] Key type lgssc registered [ 4149.554458] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4158.406469] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4158.435742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4160.068799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4164.484776] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4170.235123] LustreError: 141931:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4170.257176] LustreError: 141931:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4173.944148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4174.408055] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4178.893601] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4188.042080] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4193.949420] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4201.561915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4202.046959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4205.121665] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4208.207261] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4214.991226] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4218.435457] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4224.550829] Lustre: 144510:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4227.762817] Lustre: 144559:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4229.206600] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4230.271950] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4234.874819] Lustre: 144706:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4235.800728] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4236.801928] Lustre: 144814:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4243.947990] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4243.951835] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4243.956729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4243.993068] Lustre: Skipped 1 previous similar message [ 4248.285205] Lustre: server umount lustre-OST0000 complete [ 4254.179088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4254.185495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4254.201106] Lustre: Skipped 1 previous similar message [ 4254.215108] Lustre: Skipped 1 previous similar message [ 4259.080382] Lustre: server umount lustre-MDT0000 complete [ 4259.300869] LustreError: 141931:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4259.318375] LustreError: 141931:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4263.101279] LustreError: 141911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787123623 with bad export cookie 2731544226985362021 [ 4263.112480] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4263.127680] LustreError: 141911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4263.519206] Lustre: server umount lustre-MDT0001 complete [ 4276.288988] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 4279.990965] Key type lgssc unregistered [ 4280.381591] LNet: 145867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4280.405158] LNetError: 145867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4280.418631] LNet: Removed LNI 192.168.204.142@tcp [ 4281.710499] Key type .llcrypt unregistered [ 4281.721403] Key type ._llcrypt unregistered [ 4307.465208] Key type ._llcrypt registered [ 4307.468454] Key type .llcrypt registered [ 4307.622490] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 4308.638661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4308.850937] alg: No test for adler32 (adler32-zlib) [ 4309.964672] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 4310.285759] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 4312.040191] Key type lgssc registered [ 4313.218382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4320.609086] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4320.652491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4322.066620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4326.665936] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4331.493347] LustreError: 147280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4331.507349] LustreError: 147280:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4335.332026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4335.895316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4341.035338] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4348.706408] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4355.087287] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4363.543130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4364.149795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4367.237657] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4371.122461] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4380.261592] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4380.618474] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4386.893639] Lustre: *** cfs_fail_loc=159, val=0*** [ 4403.451834] Lustre: lustre-MDT0000: Client d268c163-f071-4d0e-ac7e-09f8414c5b07 (at 192.168.204.42@tcp) reconnecting [ 4408.812826] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4408.817371] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4408.828227] LustreError: Skipped 2 previous similar messages [ 4408.863226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4410.614820] Lustre: server umount lustre-OST0000 complete [ 4416.490483] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4416.491802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4416.519304] Lustre: Skipped 1 previous similar message [ 4416.536273] Lustre: Skipped 1 previous similar message [ 4421.063391] Lustre: server umount lustre-MDT0000 complete [ 4421.603062] LustreError: 147280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4421.628661] LustreError: 147280:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4424.789755] LustreError: 147261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787123785 with bad export cookie 15320395713897523397 [ 4424.790711] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4424.799450] LustreError: 147261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4434.561624] Lustre: server umount lustre-MDT0001 complete [ 4445.232850] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 4448.291803] Key type lgssc unregistered [ 4448.617646] LNet: 150947:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4448.626248] LNetError: 150947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4448.653537] LNet: Removed LNI 192.168.204.142@tcp [ 4449.609110] Key type .llcrypt unregistered [ 4449.611419] Key type ._llcrypt unregistered [ 4473.613321] Key type ._llcrypt registered [ 4473.618173] Key type .llcrypt registered [ 4473.715112] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 4474.689605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4474.723862] alg: No test for adler32 (adler32-zlib) [ 4475.807167] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 4476.007019] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 4477.648266] Key type lgssc registered [ 4478.916318] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4487.278645] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4487.304530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4488.909340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4493.690774] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4498.431825] LustreError: 152364:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4498.470597] LustreError: 152364:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4503.147045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4503.714894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4508.390530] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4518.015371] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4523.829989] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4532.579406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4533.007804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4534.094164] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4538.996692] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4546.095445] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4548.417187] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4570.731406] Lustre: lustre-MDT0001: evicting 975f4001-99be-42a3-991c-1fdcacc36a69 (at 192.168.204.42@tcp) by administrative request [ 4570.749244] Lustre: Skipped 1 previous similar message [ 4604.898565] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4604.909154] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4604.912433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4604.925014] Lustre: Skipped 1 previous similar message [ 4604.937742] Lustre: Skipped 1 previous similar message [ 4610.020976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4610.028114] Lustre: Skipped 1 previous similar message [ 4615.136493] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4615.153933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4615.164935] Lustre: Skipped 1 previous similar message [ 4615.308886] Lustre: server umount lustre-OST0000 complete [ 4620.267503] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4620.270897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4620.280640] Lustre: Skipped 1 previous similar message [ 4620.292613] Lustre: Skipped 1 previous similar message [ 4625.378233] LustreError: 152361:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4625.388256] LustreError: 152361:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4625.401339] Lustre: server umount lustre-MDT0000 complete [ 4629.838183] LustreError: 152344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787123990 with bad export cookie 9438191989153529738 [ 4629.859608] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4630.136574] Lustre: server umount lustre-MDT0001 complete [ 4645.310342] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 4649.758244] Key type lgssc unregistered [ 4650.216241] LNet: 156438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4650.227726] LNetError: 156438:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4650.270591] LNet: Removed LNI 192.168.204.142@tcp [ 4651.682298] Key type .llcrypt unregistered [ 4651.689467] Key type ._llcrypt unregistered [ 4724.558784] Key type ._llcrypt registered [ 4724.561347] Key type .llcrypt registered [ 4724.714150] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 4726.107747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4726.223037] alg: No test for adler32 (adler32-zlib) [ 4727.403731] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 4727.693158] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 4729.376390] Key type lgssc registered [ 4730.584559] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4738.465796] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4738.489688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4740.145594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4744.695857] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4750.308849] LustreError: 158930:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4750.342666] LustreError: 158930:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4752.671701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4753.050870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4757.184589] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4764.008924] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4769.261368] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4775.756556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4776.045083] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4778.107580] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 4781.658569] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4785.269947] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 4787.895297] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4798.946753] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4798.955059] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4798.964427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4800.997785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4800.998647] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4801.021859] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4802.288661] Lustre: server umount lustre-OST0000 complete [ 4806.115957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4806.131055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4806.136605] Lustre: Skipped 2 previous similar messages [ 4811.236561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4811.244199] Lustre: Skipped 1 previous similar message [ 4811.916996] Lustre: server umount lustre-MDT0000 complete [ 4815.480232] LustreError: 158910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787124176 with bad export cookie 1699757839669932710 [ 4815.486973] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4815.500695] LustreError: 158910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4816.033898] Lustre: server umount lustre-MDT0001 complete [ 4826.045353] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 4828.611462] Key type lgssc unregistered [ 4828.950963] LNet: 162474:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4828.962093] LNetError: 162474:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4828.977870] LNet: Removed LNI 192.168.204.142@tcp [ 4829.780130] Key type .llcrypt unregistered [ 4829.786851] Key type ._llcrypt unregistered [ 4854.558340] Key type ._llcrypt registered [ 4854.565559] Key type .llcrypt registered [ 4854.667947] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 4855.608271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4855.645770] alg: No test for adler32 (adler32-zlib) [ 4856.922685] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 4857.196433] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 4858.896202] Key type lgssc registered [ 4860.234091] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4875.218297] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 03:23:54 (1787124234) [ 4891.941838] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 4903.383365] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 4939.737325] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4939.772168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4941.064472] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4941.129979] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4941.269940] Lustre: lustre-MDT0000: new disk, initializing [ 4941.418504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4941.444075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4945.767440] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4957.031500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4957.202874] Lustre: 166737:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4957.291872] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4957.301890] Lustre: Skipped 1 previous similar message [ 4957.436920] Lustre: lustre-MDT0001: new disk, initializing [ 4957.577536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4957.631980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4957.647244] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4962.704846] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4974.031487] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4980.692420] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4989.134473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4989.507317] Lustre: lustre-OST0000: new disk, initializing [ 4989.512416] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4989.519945] Lustre: 168705:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4989.610523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4996.188198] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4997.727345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4997.738071] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4997.804552] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5006.296451] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5018.594845] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5018.608617] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5018.617197] LustreError: Skipped 1 previous similar message [ 5018.638903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5018.646711] Lustre: Skipped 1 previous similar message [ 5018.683177] Lustre: Skipped 1 previous similar message [ 5021.603259] Lustre: server umount lustre-OST0000 complete [ 5028.838467] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5028.843281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5028.857110] Lustre: Skipped 1 previous similar message [ 5028.874437] Lustre: Skipped 1 previous similar message [ 5031.830389] Lustre: server umount lustre-MDT0000 complete [ 5036.024264] LustreError: 169659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787124396 with bad export cookie 15533561528903724338 [ 5036.030499] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5036.036304] LustreError: 169659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5036.427712] Lustre: server umount lustre-MDT0001 complete [ 5047.486795] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 5050.386233] Key type lgssc unregistered [ 5050.734507] LNet: 170615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5050.745783] LNetError: 170615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5050.770272] LNet: Removed LNI 192.168.204.142@tcp [ 5051.894334] Key type .llcrypt unregistered [ 5051.897433] Key type ._llcrypt unregistered [ 5079.294487] Key type ._llcrypt registered [ 5079.296590] Key type .llcrypt registered [ 5079.375566] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 5080.162916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5080.286944] alg: No test for adler32 (adler32-zlib) [ 5081.385273] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 5081.668199] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 5083.400181] Key type lgssc registered [ 5084.610655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5092.235087] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5092.259277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5093.738261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5097.733427] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5104.101987] LustreError: 172031:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5104.118907] LustreError: 172031:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5105.135335] LustreError: 172032:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5106.666249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5107.015614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5111.998658] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5120.562444] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5127.197272] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5134.736652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5135.334710] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5139.444312] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5141.928168] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5149.988792] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5159.486375] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5159.502414] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5159.517463] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5159.528313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5159.550404] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5160.374460] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5160.384877] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5160.395192] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5160.402769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5160.417102] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5161.591643] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5161.620318] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5161.628966] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5161.634668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5161.642875] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5162.470988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5163.390710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5164.104788] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5164.110175] Lustre: Skipped 2 previous similar messages [ 5164.123798] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5164.132160] Lustre: Skipped 2 previous similar messages [ 5164.136456] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5164.140204] Lustre: Skipped 2 previous similar messages [ 5164.143355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5164.153754] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5164.162148] Lustre: Skipped 2 previous similar messages [ 5164.928622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5165.676503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5166.417694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5167.187375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5167.983477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5168.673432] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5168.679538] Lustre: Skipped 5 previous similar messages [ 5168.702850] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5168.718438] Lustre: Skipped 5 previous similar messages [ 5168.724127] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5168.732092] Lustre: Skipped 5 previous similar messages [ 5168.736310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5168.756836] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5168.766859] Lustre: Skipped 5 previous similar messages [ 5169.625211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5170.550731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5171.446628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5172.251827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5173.253815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5174.228048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5175.192553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5175.993847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5177.151064] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5177.163761] Lustre: Skipped 8 previous similar messages [ 5177.197177] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5177.205898] Lustre: Skipped 8 previous similar messages [ 5177.211962] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5177.216459] Lustre: Skipped 8 previous similar messages [ 5177.220563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5177.239196] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5177.255956] Lustre: Skipped 8 previous similar messages [ 5177.991939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5178.968234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5180.173452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5181.242373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5182.359532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5183.525871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5184.772558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5185.777247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5186.801341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5188.181426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5189.357936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5190.500962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5191.336317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5192.225907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5193.398698] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5193.408736] Lustre: Skipped 14 previous similar messages [ 5193.446580] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5193.459066] Lustre: Skipped 14 previous similar messages [ 5193.472272] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5193.478178] Lustre: Skipped 14 previous similar messages [ 5193.486299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5193.506506] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5193.520391] Lustre: Skipped 14 previous similar messages [ 5194.670247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5196.025827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5197.217679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5198.355931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5199.240869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5200.188576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5201.069475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5202.016128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5202.806759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5203.702113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5204.619338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5205.482392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5206.336501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5207.288853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5208.152643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5209.005364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5209.858446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5210.969342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5211.796520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5212.632124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5213.717938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5214.674353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5215.597586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5216.587393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5217.577762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5218.350964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5219.195436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5220.083567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5220.935621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5221.992510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5222.736134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5223.545658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5224.270726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5225.031428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5225.853293] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5225.860491] Lustre: Skipped 34 previous similar messages [ 5225.874159] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5225.891255] Lustre: Skipped 34 previous similar messages [ 5225.899490] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5225.908130] Lustre: Skipped 34 previous similar messages [ 5225.913554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5225.927362] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5225.932790] Lustre: Skipped 34 previous similar messages [ 5226.784018] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5227.577994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5228.456906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5229.235739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5230.014967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5230.816653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5231.696706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5232.471399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5233.459045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5234.254084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5235.045718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5235.846957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5236.637790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5237.403490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5238.211350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5239.036623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5240.053891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5240.850911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5241.535090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5242.490702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5243.299639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5244.203587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5245.008290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5245.871478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5246.841326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5247.952318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5248.777582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5249.700065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5250.619551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5251.574202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5252.449432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5253.458724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5254.298329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5255.170634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5256.140730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5257.160640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5258.017330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5258.905135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5259.712952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5260.574792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5261.460146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5262.544993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5263.562291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5264.683420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5265.547702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5266.556313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5267.881765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5268.933993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5269.896738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5270.797726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5271.804391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5272.683210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5273.632224] LustreError: 172051:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5273.647074] LustreError: 172051:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5273.658130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5274.511648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5275.416341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5276.527394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5277.271886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5278.166968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5279.026129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5280.102630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5280.989821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5281.998143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5283.375946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5284.693046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5285.857277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5286.907591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5287.943477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5288.892935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5289.992285] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5290.004310] Lustre: Skipped 68 previous similar messages [ 5290.027787] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5290.039833] Lustre: Skipped 68 previous similar messages [ 5290.044256] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5290.050498] Lustre: Skipped 68 previous similar messages [ 5290.058128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5290.068942] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5290.080233] Lustre: Skipped 68 previous similar messages [ 5291.025581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5292.000933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5292.926051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5293.855639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5294.900635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5295.776650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5296.866897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5298.071467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5298.985604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5300.067698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5301.123583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5301.904723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5303.111170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5304.192091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5305.355372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5306.759061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5308.402457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5309.365518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5310.911797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5312.358971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5313.321753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5314.688710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5316.051499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5317.326859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5318.256697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5319.445751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5320.630092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5322.016900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5323.324328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5324.601398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5326.003044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5327.204894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5328.200643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5329.131537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5330.103400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5331.193585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5332.250235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5333.106273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5334.266960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5335.123229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5336.061373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5336.928635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5337.884784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5338.792747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5339.774059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5340.841240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5341.850576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5342.676675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5343.633730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5344.619668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5345.546237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5346.652821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5347.750225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5348.854168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5349.870193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5350.977933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5351.930734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5353.046352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5353.839197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5354.997703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5356.159414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5357.379545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5358.485571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5359.620963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5360.599026] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5362.002650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5363.442579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5364.489290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5365.713952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5367.365526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5368.521535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5369.985905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5371.154836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5372.182158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5373.189422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5374.166027] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5375.570608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5376.792546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5378.360738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5379.587214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5380.662335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5381.760950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5382.723418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5383.803614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5384.724734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5385.709827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5386.704616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5387.841813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5388.900736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5389.915815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5390.915772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5392.043395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5393.392371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5394.357523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5395.322783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5396.406281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.623940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5398.602144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5399.606906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5400.452578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5401.536196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5402.739665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5403.877229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5404.975892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5406.019981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5407.090312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.143528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5409.211537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5410.426238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5411.493441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5412.423906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5413.477941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5414.523324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5415.629489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5416.625301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5417.650274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.882243] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5418.890015] Lustre: Skipped 116 previous similar messages [ 5418.911036] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5418.921206] Lustre: Skipped 116 previous similar messages [ 5418.926024] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5418.931418] Lustre: Skipped 116 previous similar messages [ 5418.938549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.953785] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5418.958723] Lustre: Skipped 116 previous similar messages [ 5419.879291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5421.003240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5422.032158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5423.214908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5424.772627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5426.142606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5427.461733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5428.509800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5429.452914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5430.551424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5431.660841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5432.619404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5433.857669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5434.845612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5436.114474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5437.528334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5438.667515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5439.847938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5441.140209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5442.428573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5443.487495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5444.506446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5445.617362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5446.543850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5447.572702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5448.767574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5449.863267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5450.859053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5452.079691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5453.217444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5454.083376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5455.100505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5456.165676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5457.299390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5458.147849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5459.305064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5460.280590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.412058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5462.357250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5463.637864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5464.846546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5466.048940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5466.952732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.918517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5468.755793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.878564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5471.004681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5472.230793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.339626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5474.533110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.777549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5477.118830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5478.468905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5479.620400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5480.809403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.819363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5483.089729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.241322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.486029] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5486.882137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.283865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.499616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5490.702243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.827126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.809371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.858587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5494.729342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.933062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5496.902495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.920166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.316148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.395944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.450461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5502.661761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5503.972668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.171598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.477539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.717263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.767914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.058970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5511.109273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.181681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.345128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5514.534844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.579345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.108858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.914447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.430088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.753336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.294993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.545167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5526.046759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.338803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.888049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5530.052265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.165992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.312982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5533.544095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.566246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5535.854762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.716819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5537.798993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.676442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.002043] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5541.304205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.374276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.444276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.659769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.932114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.057630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.309030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.404730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.566622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.736936] LustreError: 172051:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5551.748879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.755628] LustreError: 172051:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5553.041179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.244744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.529231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5556.594779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.388623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.580054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.658780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.890285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.965793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.993879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.116443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5565.267346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.453729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.299498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.262336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.324621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.282520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.422410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.468882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.568194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.058129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.284642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.472916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.545281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.486697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.920031] LustreError: 172051:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5580.932386] LustreError: 172051:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5580.941290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.204139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.182749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.300417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.552622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.018329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.163086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.309877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.547455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.316835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.568213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.012226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.586826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.007996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.617470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.878676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.214750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.416430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.242904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.291210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.621560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.612072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.491411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.543723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.366871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.332945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.425524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.555824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.651491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.544587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.613260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.879969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.938918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.071642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.954522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.910061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.839179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.020386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.792765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.089576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.175753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.304646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.643968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.589329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.775871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.157063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.396820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.445881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.434913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.533907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.411889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.504071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.513306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.671902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.797725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.823465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.054569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.100293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.115744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.117839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.113273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.177662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.660709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.607347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.743846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.535732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.771676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.803446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.625726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.744438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.780153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.110692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.556527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.151957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.446038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.094802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.039522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.347497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.900785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.477593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.894570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.234929] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5675.241153] Lustre: Skipped 220 previous similar messages [ 5675.262819] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5675.271709] Lustre: Skipped 220 previous similar messages [ 5675.274840] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5675.278976] Lustre: Skipped 220 previous similar messages [ 5675.282669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.289572] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5675.295303] Lustre: Skipped 220 previous similar messages [ 5676.332767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.646808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.910360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.061669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.999576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.081790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.905436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.742556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.664566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.634657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.448537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.309215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.139808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.993943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.845797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.831984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.601697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.542822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.570714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.613050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.544779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.564046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.628552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.674656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.714771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.655407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.874864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.874691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.854920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.869354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.934821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.096326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.145791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.056495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.040697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.942913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.865392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5712.765770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.696645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.656741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.602634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.444519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.331671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.256284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.213665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.106997] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.961675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.945783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.913730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.835974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.898533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.763878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.796932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.763674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.608775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.608914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.727332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.728273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.836499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.783721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.907751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.782388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.748607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.878287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.880537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.962035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.078270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.847244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.812501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.701603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.686322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.827340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.786380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.707302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.578168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.659214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5750.610706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.681672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.824933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.910652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.190897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.380628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.496944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.591410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.631177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.718922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.652952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.651890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.941642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.316870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.295256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.299393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.176182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.052737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.015599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.248126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.204831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.205539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.324519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.603658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.773356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.000750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.085842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.128764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.074364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.975087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.852151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.776567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.596769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.594561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.445941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.380766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.193769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.263116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.227750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.168074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.296961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.314581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.429552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.308473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.362809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.226996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.305720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.334284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.410130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.436503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.367733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.246682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.210090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.215831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.199881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.064243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.921318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.648133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.644154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.464321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.473739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.407955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.240652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.944579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5814.953431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.008761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.839878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.818293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.925269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.719050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.806186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.743126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.826778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.897696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.968466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.204953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.067648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.294196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.172389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.208398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.432844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.510466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.930561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5835.246683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.190871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.937351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.957843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.182203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.573298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.961822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.842562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.791570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5844.597412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.319477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.716492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.111298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.653692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.332264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.918312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.164680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.480737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5857.931536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.318448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.595930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.935859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.973912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.127726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.057777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.098510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.082537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.118763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.105431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.164048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.415378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.581514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.786370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.867878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.187928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.281982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.422091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.366783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.523281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5881.676673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5882.789277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.812925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.804986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.810942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.942263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.803639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.940191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.126476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.134841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.226985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.390923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.462453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.416536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.531439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.465206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.586894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.661765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.569940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5901.527793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.524220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.477347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.596930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.635045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.793589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.935873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.001237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.038346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.128818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.170046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.329843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5914.326205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.313611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.257863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.249476] LustreError: 172051:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5917.268490] LustreError: 172051:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5917.275713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.292164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5919.419024] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.439546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5921.453900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.642479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5923.729947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.790847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.681893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.810471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5927.788337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5928.934287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5929.960276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5930.929224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.719395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.672690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5933.523112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5934.538745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5935.616247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5936.566355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5937.540247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5938.401394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.465771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.444332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5941.591439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.684903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5943.814292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5944.834835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5945.724789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.638222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5947.526909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.358717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.326805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5950.253961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5951.139050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5952.167589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.494892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5954.591195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5955.744690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5956.790368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5958.054194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5959.348891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5960.583473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5961.501969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5962.555991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5963.576641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5964.542481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5965.554476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5966.442382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.412795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5968.430905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5969.226719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5970.179322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5971.033933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5972.054251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5972.992794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5974.038693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.045519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.066603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.966169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.071990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.958107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5980.288679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5981.367111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5982.712732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5983.721837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.579703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5985.580497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.589588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5987.559824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.585573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5989.328529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5990.105235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.011157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.967889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5992.729290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5993.597689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5994.588451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5995.689156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5996.650079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5997.648627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.504824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5999.694708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.649107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6001.738659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6002.785578] LustreError: 172051:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6002.794706] LustreError: 172051:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6002.810712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6003.899153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6004.772069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6005.437859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.353957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6007.347114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6008.376770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6009.340874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6010.472191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6011.589149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6012.618472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6013.730330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.809588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6016.009271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6016.943894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6017.969614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6018.924471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6019.903653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6020.818924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6021.814867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6022.766098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6023.928883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6024.834800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6025.590710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6026.508785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6027.439415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6028.499224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6029.398212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6030.656168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6031.870774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6032.858442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6033.781519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.719952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6035.846180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6036.874450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6037.863510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.922769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6039.960282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6040.897578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6041.951951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6042.991343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6043.878486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6044.938291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6045.878696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6046.877424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6047.793856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6048.773280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6049.728071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6050.608164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6051.545262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6052.321306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6053.283445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6054.256903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6055.139935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.060032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.935157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6057.874934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6058.824790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6059.774242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6060.581221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6061.660956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6062.750509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6063.806284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.906500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6065.804249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6066.911219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6067.810280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6068.722214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6069.873151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6071.081382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6072.131696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6072.928186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6073.754125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6074.857832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.971674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6077.049673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6077.983739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6079.046634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6080.060625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6081.098348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6082.233305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6083.590166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6084.709882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6085.677881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6086.560913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6087.698560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6088.663298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6089.771810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6090.748302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6091.705381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6092.681738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6093.902826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6095.015713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6096.176163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6097.491335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6098.529522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6099.454371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6100.501894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6101.845142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6103.161979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6104.224224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6105.337639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6106.470126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6107.500143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6108.502231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6109.446953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6110.476631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6111.590869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6112.658766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6113.665240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6114.695916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6115.781953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6116.945592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6118.099219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6118.998996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6120.239680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6121.349365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6122.412400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6123.370841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6124.486683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6125.505660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6126.447439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6127.413549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6128.288266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6129.326251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6130.439059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6131.347231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6132.322265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6133.268799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6134.318336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6135.206049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6136.462501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6137.428424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6138.824516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6139.954837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6141.326425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6142.848824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6143.802916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6144.936780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6146.072445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6147.275586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6148.268597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6149.385202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6150.398122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6151.555405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6152.770257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6153.742161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6154.912740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6155.885402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6157.001151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6157.930370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6159.118369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6160.310322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6161.466687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6162.771987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6164.032302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6165.001121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6165.993527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6167.045931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6168.365395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6169.713576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6171.338019] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6172.317082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6173.456882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6174.578160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6175.649163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6176.645233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6177.589060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6178.548791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6179.620283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6180.677030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6181.656378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6182.637612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6183.672443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6184.626886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6185.599318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6186.905206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6188.231898] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6188.237067] Lustre: Skipped 494 previous similar messages [ 6188.262671] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6188.271475] Lustre: Skipped 494 previous similar messages [ 6188.275411] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6188.277657] Lustre: Skipped 494 previous similar messages [ 6188.281750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6188.289895] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6188.295156] Lustre: Skipped 494 previous similar messages [ 6189.139866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6190.102222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6190.983813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6192.125940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6193.303324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6194.366174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6195.685445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6196.919987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6198.078309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6198.970689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6199.980892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6200.786598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6201.739210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6202.577636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6203.482939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6204.388273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6205.451496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6206.573210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6207.397273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6208.475821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6209.590635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6210.576325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6211.456455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6212.468894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6213.343208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6214.240628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6215.159270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6611.498790] LustreError: 173855:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6611.518056] LustreError: 172051:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 7256.171155] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7256.171945] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7256.176079] LustreError: Skipped 2 previous similar messages [ 7256.189219] Lustre: Skipped 27 previous similar messages [ 7256.197062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7258.085835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7258.101530] Lustre: Skipped 1 previous similar message [ 7258.442915] Lustre: server umount lustre-OST0000 complete [ 7262.690309] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7262.700593] LustreError: Skipped 1 previous similar message [ 7262.708524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7268.322552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7268.332365] Lustre: Skipped 3 previous similar messages [ 7268.907730] Lustre: server umount lustre-MDT0000 complete [ 7273.441362] LustreError: 172028:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7273.490799] LustreError: 172028:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7273.766928] LustreError: 172012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787126634 with bad export cookie 14252255572756847042 [ 7273.778048] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7274.271368] Lustre: server umount lustre-MDT0001 complete [ 7286.170810] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 7289.511052] Key type lgssc unregistered [ 7289.864382] LNet: 227320:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7289.869743] LNetError: 227320:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7289.890861] LNet: Removed LNI 192.168.204.142@tcp [ 7291.105537] Key type .llcrypt unregistered [ 7291.107891] Key type ._llcrypt unregistered [ 7320.389944] Key type ._llcrypt registered [ 7320.392181] Key type .llcrypt registered [ 7320.487404] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 7321.397288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7321.477067] alg: No test for adler32 (adler32-zlib) [ 7322.583101] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 7322.894514] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 7324.592228] Key type lgssc registered [ 7325.804487] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7333.609712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7333.637902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7335.093892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7339.460603] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7344.614913] LustreError: 228740:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7344.647255] LustreError: 228740:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7348.331354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7348.632142] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7353.887489] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7361.624158] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7367.613544] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7375.531872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7376.040973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7380.160674] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 7383.640418] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7391.490402] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7397.948655] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7400.647432] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7400.673474] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 7400.686651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7400.701886] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7400.705787] LustreError: 228759:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 7404.001401] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7404.012205] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7404.044290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7406.050036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7406.055952] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7406.057031] Lustre: Skipped 1 previous similar message [ 7409.365762] Lustre: lustre-OST0000: Not available for connect from 192.168.204.42@tcp (stopping) [ 7409.379328] Lustre: Skipped 1 previous similar message [ 7409.675667] Lustre: server umount lustre-OST0000 complete [ 7420.800834] LustreError: 228759:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 7445.245418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7445.537345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7446.580616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7446.606816] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7451.988134] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7455.862108] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7455.880692] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7459.655824] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7461.058839] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7466.190745] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8cf245829800.ost_server_uuid 50 [ 7467.746894] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8cf245829800.ost_server_uuid in FULL state after 0 sec [ 7473.121712] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7473.137910] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7473.159163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7474.718746] Lustre: server umount lustre-OST0000 complete [ 7481.826101] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7481.828288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7481.847784] Lustre: Skipped 1 previous similar message [ 7481.880434] Lustre: Skipped 1 previous similar message [ 7492.071702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7492.077096] Lustre: Skipped 3 previous similar messages [ 7493.089084] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7493.089471] Lustre: 228052:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787126837/real 1787126837] req@0000000038b66de5 x1873938148829952/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787126853 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7493.090485] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7493.159081] Lustre: 228052:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7493.384891] Lustre: server umount lustre-MDT0000 complete [ 7497.583938] LustreError: 228719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787126858 with bad export cookie 4314599696254894076 [ 7497.593605] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7497.602204] LustreError: 228719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7497.887517] Lustre: server umount lustre-MDT0001 complete [ 7508.764783] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 7511.785987] Key type lgssc unregistered [ 7512.104529] LNet: 233958:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7512.114691] LNetError: 233958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7512.127240] LNet: Removed LNI 192.168.204.142@tcp [ 7513.179865] Key type .llcrypt unregistered [ 7513.184751] Key type ._llcrypt unregistered [ 7551.183773] Key type ._llcrypt registered [ 7551.186733] Key type .llcrypt registered [ 7551.328050] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 7552.865033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7552.897157] alg: No test for adler32 (adler32-zlib) [ 7553.969366] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 7554.332293] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 7556.081226] Key type lgssc registered [ 7557.328895] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7566.780935] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7566.831418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7568.521516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7573.566611] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7579.115192] LustreError: 235691:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7579.134916] LustreError: 235691:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7583.404697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7583.939303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7588.943401] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7592.384120] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7599.654959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7600.163277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7607.982778] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7613.504445] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 7617.828658] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7617.994927] Lustre: lustre-OST0001: new disk, initializing [ 7618.000124] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7618.008944] Lustre: 238000:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7618.108799] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7623.793183] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7627.835444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7627.853692] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7627.975511] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7636.272570] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7642.574434] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7658.985121] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7659.000338] Lustre: Skipped 1 previous similar message [ 7659.004263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7659.025410] Lustre: Skipped 3 previous similar messages [ 7664.102079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7664.107979] Lustre: Skipped 3 previous similar messages [ 7664.994909] Lustre: server umount lustre-MDT0000 complete [ 7668.852823] LustreError: 235671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787127029 with bad export cookie 3111765032924465479 [ 7668.856748] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7668.863292] LustreError: 235671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7669.166645] Lustre: server umount lustre-MDT0001 complete [ 7683.977372] Lustre: server umount lustre-OST0000 complete [ 7685.095798] Lustre: 234952:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787127029/real 1787127029] req@ffff8c5283321500 x1873938392118912/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787127045 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7685.126406] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7685.139434] Lustre: Skipped 1 previous similar message [ 7687.406896] Lustre: server umount lustre-OST0001 complete [ 7708.341601] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 7718.308888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7718.749777] LustreError: 241104:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7718.845101] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7723.198146] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7724.006802] LustreError: 241105:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7729.125939] LustreError: 241104:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7732.195541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7732.661575] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7737.046050] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7740.254677] Lustre: 242247:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7747.450986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7754.042266] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7758.117756] LustreError: 242603:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7758.146828] Lustre: mylustre-OST0000: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:100 to 0x280000400:129) [ 7763.225189] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7763.584217] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7763.591929] Lustre: Skipped 1 previous similar message [ 7764.634940] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49760) [ 7770.451622] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7778.541505] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7782.857608] Lustre: 244118:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7799.270520] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7799.288247] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7799.309525] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7803.510979] Lustre: server umount mylustre-MDT0000 complete [ 7805.929532] LustreError: 241838:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7805.947185] LustreError: 241838:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7807.777324] LustreError: 241080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787127168 with bad export cookie 3111765032924468335 [ 7807.778434] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7807.793269] LustreError: 241080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7808.138917] Lustre: server umount mylustre-MDT0001 complete [ 7822.806773] Lustre: server umount mylustre-OST0000 complete [ 7826.400154] Lustre: 234954:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787127171/real 1787127171] req@ffff8c52a3e58700 x1873938392183168/t0(0) o400->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787127187 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7826.434792] Lustre: mylustre-MDT0001-lwp-OST0001: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7826.454210] Lustre: Skipped 3 previous similar messages [ 7826.995496] Lustre: server umount mylustre-OST0001 complete [ 7851.167850] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 7862.989513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7863.550048] LustreError: 246427:0:(ldlm_lib.c:1192:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7863.721788] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7870.286591] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7882.393427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7888.740750] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7892.806566] Lustre: 247569:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7901.154446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7907.756120] LustreError: 247921:0:(ldlm_lib.c:1192:target_handle_connect()) tfs-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7907.767270] Lustre: tfs-OST0000: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7907.779380] LustreError: 247921:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7908.781830] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49792) [ 7909.210798] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7919.425638] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7927.894742] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7936.132773] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7940.157425] Lustre: 249443:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7949.282517] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7949.288503] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7949.317821] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 7949.328651] Lustre: Skipped 4 previous similar messages [ 7954.385834] Lustre: server umount tfs-MDT0000 complete [ 7959.085064] LustreError: 246402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787127319 with bad export cookie 3111765032924470295 [ 7959.102669] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7959.109414] LustreError: 246402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7959.514086] Lustre: server umount tfs-MDT0001 complete [ 7973.583060] Lustre: server umount tfs-OST0000 complete [ 7976.481163] Lustre: 234952:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787127321/real 1787127321] req@ffff8c53bdef8a80 x1873938392237824/t0(0) o400->tfs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787127337 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7976.506313] Lustre: tfs-MDT0001-lwp-OST0001: Connection to tfs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7976.520700] Lustre: Skipped 3 previous similar messages [ 7977.898441] Lustre: server umount tfs-OST0001 complete [ 8001.006515] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 8012.637870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8013.108540] LustreError: 251692:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8013.132734] LustreError: 251692:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 8013.218724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8013.226389] Lustre: Skipped 3 previous similar messages [ 8018.249592] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8028.090141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8034.599399] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8037.950806] Lustre: 252834:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8046.013096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8053.695441] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 8053.702958] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8057.858073] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49824) [ 8062.937805] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8069.782177] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8077.967840] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8081.975585] Lustre: 254703:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8089.183491] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 04:17:28 (1787127448) [ 8109.550638] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8109.556810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8109.563635] Lustre: Skipped 2 previous similar messages [ 8109.579769] Lustre: Skipped 7 previous similar messages [ 8112.349119] Lustre: server umount lustre-MDT0000 complete [ 8116.536320] LustreError: 251669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787127477 with bad export cookie 3111765032924471989 [ 8116.543367] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8116.559106] LustreError: 251669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8119.787361] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8119.796298] Lustre: Skipped 1 previous similar message [ 8123.124768] Lustre: server umount lustre-MDT0001 complete [ 8141.280142] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8141.387431] Lustre: server umount lustre-OST0000 complete [ 8151.667774] Lustre: server umount lustre-OST0001 complete [ 8157.626733] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 8168.982307] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 8208.395128] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 8208.395128] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 8208.395128] [ 8208.489327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 8209.110662] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8209.224790] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8209.489775] Lustre: lustre-MDT0000: new disk, initializing [ 8209.662040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8209.675897] Lustre: Skipped 3 previous similar messages [ 8209.739324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8216.243625] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8229.271632] LDISKFS-fs (dm-1): Mount option "noacl" will be removed by 3.5 [ 8229.271632] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 8229.271632] [ 8229.305471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 8229.478702] Lustre: 257891:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8229.556718] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8229.562653] Lustre: Skipped 1 previous similar message [ 8229.655469] Lustre: lustre-MDT0001: new disk, initializing [ 8229.820747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8229.830732] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8235.557951] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8247.145896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8247.592918] Lustre: lustre-OST0000: new disk, initializing [ 8247.598347] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8247.610057] Lustre: 259477:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8248.941867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8248.950642] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8249.094463] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8253.947885] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8266.343535] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8266.636653] Lustre: lustre-OST0001: new disk, initializing [ 8266.647419] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8266.661974] Lustre: 260347:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8268.028027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8268.049385] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8268.201709] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8274.099813] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8285.809658] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 04:20:45 (1787127645) [ 8300.663560] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 04:21:00 (1787127660) [ 8315.873473] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8315.873851] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8315.887262] LustreError: Skipped 1 previous similar message [ 8315.908066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8315.917203] Lustre: Skipped 4 previous similar messages [ 8315.932159] Lustre: Skipped 3 previous similar messages [ 8321.859585] Lustre: server umount lustre-OST0000 complete [ 8324.593991] LustreError: 259461:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8324.615581] LustreError: 259461:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 8327.140412] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8332.485746] Lustre: server umount lustre-MDT0000 complete [ 8335.943851] LustreError: 257883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787127696 with bad export cookie 3111765032924473543 [ 8335.949608] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8335.958958] LustreError: 257883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8342.375412] Lustre: server umount lustre-MDT0001 complete [ 8351.842829] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 8421.332809] Lustre: server umount lustre-OST0001 complete [ 8427.671936] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 8441.994829] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 8473.812706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8474.030767] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8474.060524] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8474.117645] Lustre: lustre-MDT0000: new disk, initializing [ 8474.172607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8474.180734] Lustre: Skipped 3 previous similar messages [ 8474.190880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8478.206875] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8488.596772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8488.707380] Lustre: 265140:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8489.005156] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8493.699646] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8502.588525] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8508.094366] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8514.314110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8514.527442] Lustre: lustre-OST0000: new disk, initializing [ 8514.531829] Lustre: Skipped 1 previous similar message [ 8514.535665] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8514.538410] Lustre: Skipped 2 previous similar messages [ 8514.542610] Lustre: 267107:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8516.226915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8516.236288] Lustre: Skipped 1 previous similar message [ 8516.243327] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8516.332890] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8519.948571] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8528.962717] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8537.067502] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8537.075553] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8537.084406] Lustre: Skipped 4 previous similar messages [ 8537.101955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8537.106125] LustreError: Skipped 1 previous similar message [ 8537.123714] Lustre: Skipped 9 previous similar messages [ 8542.528868] Lustre: server umount lustre-OST0000 complete [ 8552.537767] Lustre: server umount lustre-MDT0000 complete [ 8556.160893] LustreError: 265134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787127916 with bad export cookie 3111765032924475013 [ 8556.171154] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8556.176675] LustreError: 265134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8556.464735] Lustre: server umount lustre-MDT0001 complete [ 8566.768610] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 8569.162537] Key type lgssc unregistered [ 8569.449974] LNet: 268870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8569.471549] LNetError: 268870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8569.489761] LNet: Removed LNI 192.168.204.142@tcp [ 8570.314205] Key type .llcrypt unregistered [ 8570.318758] Key type ._llcrypt unregistered [ 8595.566295] Key type ._llcrypt registered [ 8595.569931] Key type .llcrypt registered [ 8595.671811] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 8596.911530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8596.955757] alg: No test for adler32 (adler32-zlib) [ 8598.004491] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 8598.241664] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 8599.904516] Key type lgssc registered [ 8601.094155] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8609.219664] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8609.240159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8610.671515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8614.781498] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8621.029868] LustreError: 270333:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8621.045046] LustreError: 270333:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8622.242584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8622.648251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8626.934632] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8633.909331] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8638.980684] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8645.332285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8645.702548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8651.671736] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8655.931179] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 8657.995573] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8658.827513] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 8660.966164] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8660.972475] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8660.974943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8660.990164] Lustre: Skipped 1 previous similar message [ 8662.635285] Lustre: server umount lustre-OST0000 complete [ 8666.084447] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8666.086562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8666.094777] Lustre: Skipped 1 previous similar message [ 8666.105551] Lustre: Skipped 1 previous similar message [ 8671.202899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8671.205464] Lustre: Skipped 2 previous similar messages [ 8672.276319] Lustre: server umount lustre-MDT0000 complete [ 8675.797905] LustreError: 272863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787128036 with bad export cookie 16043937991638120323 [ 8675.799696] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8675.807689] LustreError: 272863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8676.158770] Lustre: server umount lustre-MDT0001 complete [ 8685.940920] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 8688.677428] Key type lgssc unregistered [ 8689.012873] LNet: 273919:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8689.030708] LNetError: 273919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8689.049861] LNet: Removed LNI 192.168.204.142@tcp [ 8689.883988] Key type .llcrypt unregistered [ 8689.886560] Key type ._llcrypt unregistered [ 8706.978917] Key type ._llcrypt registered [ 8706.985605] Key type .llcrypt registered [ 8707.120803] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 8708.167273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8708.214993] alg: No test for adler32 (adler32-zlib) [ 8709.323586] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 8709.589965] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 8711.249975] Key type lgssc registered [ 8712.461931] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8721.226122] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8721.277409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8736.800451] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8741.920367] LustreError: 275116:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.142@tcp: failed processing log, type 4: rc = -110 [ 8768.608358] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8774.446608] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8781.931092] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8784.175354] Lustre: server umount lustre-OST0000 complete [ 8800.851859] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 8803.476178] Key type lgssc unregistered [ 8803.772504] LNet: 276751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8803.783828] LNetError: 276751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8803.815215] LNet: Removed LNI 192.168.204.142@tcp [ 8804.719852] Key type .llcrypt unregistered [ 8804.721495] Key type ._llcrypt unregistered [ 8844.392642] Key type ._llcrypt registered [ 8844.394283] Key type .llcrypt registered [ 8844.518853] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 8846.015661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8846.041278] alg: No test for adler32 (adler32-zlib) [ 8847.235026] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 8847.574624] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 8849.328764] Key type lgssc registered [ 8850.507816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8864.749943] loop0: detected capacity change from 0 to 204800000 [ 8864.984447] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8865.009255] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8865.387429] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8866.459523] loop0: detected capacity change from 0 to 204800000 [ 8866.604270] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8866.614123] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8867.032222] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8867.994969] loop0: detected capacity change from 0 to 204800000 [ 8868.133733] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8868.148796] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8868.260658] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8869.209972] loop0: detected capacity change from 0 to 204800000 [ 8869.450230] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8870.791486] loop0: detected capacity change from 0 to 204800000 [ 8870.834506] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8875.336742] loop0: detected capacity change from 0 to 204800000 [ 8875.375690] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8879.987133] loop0: detected capacity change from 0 to 204800000 [ 8880.039565] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8884.291149] loop0: detected capacity change from 0 to 204800000 [ 8884.330263] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8888.362475] loop0: detected capacity change from 0 to 204800000 [ 8888.468396] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8888.501035] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8888.578633] Lustre: 280165:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8888.587920] Lustre: 280165:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 8888.599165] Lustre: 280165:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8888.607303] Lustre: 280165:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 8888.615276] Lustre: 280165:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 8888.622725] Lustre: 280165:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8888.632462] Lustre: 280165:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8888.640341] Lustre: 280165:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 8888.647892] Lustre: 280165:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 8888.657289] Lustre: 280165:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 8888.665503] Lustre: 280165:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 8888.675727] Lustre: 280165:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 8888.687732] Lustre: 280165:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 8888.707367] Lustre: 280165:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 8894.785049] Lustre: server umount MGS complete [ 8895.742200] loop0: detected capacity change from 0 to 204800000 [ 8895.877952] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8896.296676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8896.344057] Lustre: lustre-MDT0000: Aborting client recovery [ 8897.372294] loop1: detected capacity change from 0 to 204800000 [ 8897.572056] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8897.679619] Lustre: 280598:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8897.695591] Lustre: 280598:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8897.710913] Lustre: 280598:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8897.736716] Lustre: 280598:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 8897.753670] Lustre: 280598:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 8897.774282] Lustre: 280598:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8897.794142] Lustre: 280598:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 8897.853669] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8898.095098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8898.182919] Lustre: lustre-MDT0001: Aborting client recovery [ 8899.236118] loop2: detected capacity change from 0 to 204800000 [ 8899.451933] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8899.569915] Lustre: 280676:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8899.581458] Lustre: 280676:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8899.590804] Lustre: 280676:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8899.601621] Lustre: 280676:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8899.612352] Lustre: 280676:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8899.622869] Lustre: 280676:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8899.640607] Lustre: 280676:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8899.829809] Lustre: Found index 0 for lustre-OST0000, updating log [ 8899.918904] Lustre: 280694:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8900.074308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8900.128881] Lustre: lustre-OST0000: Aborting recovery [ 8901.221295] loop3: detected capacity change from 0 to 204800000 [ 8901.367773] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8901.435143] Lustre: 280761:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8901.458166] Lustre: 280761:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8901.473263] Lustre: 280761:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8901.481177] Lustre: 280761:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8901.490641] Lustre: 280761:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8901.504975] Lustre: 280761:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8901.523130] Lustre: 280761:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8901.549223] Lustre: Found index 1 for lustre-OST0001, updating log [ 8901.591729] Lustre: 280767:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8901.738885] Lustre: lustre-OST0001: Aborting recovery [ 8905.828702] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8905.850340] LustreError: 280542:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -108 [ 8905.878911] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8905.889363] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 8905.945382] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 8907.885213] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 8907.885910] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 8909.615688] LustreError: 280509:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 819us: rc = -2 [ 8910.187125] LustreError: 280509:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 3366us: rc = -2 [ 8910.202921] LustreError: 280509:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 22 previous similar messages [ 8923.619750] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8923.622956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8923.629121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8923.651752] Lustre: Skipped 1 previous similar message [ 8928.752622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8928.759160] Lustre: Skipped 6 previous similar messages [ 8929.905348] Lustre: server umount lustre-MDT0000 complete [ 8930.832569] LustreError: 280492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787128291 with bad export cookie 12811114012279556749 [ 8930.851051] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8933.858325] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8933.858860] LustreError: 280508:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8933.860519] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8933.877239] Lustre: Skipped 3 previous similar messages [ 8933.912488] LustreError: 280508:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8935.905589] LustreError: 280506:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8935.905773] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8935.935698] LustreError: 280506:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8935.946885] Lustre: Skipped 2 previous similar messages [ 8941.030722] LustreError: 280508:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8941.030989] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8941.043105] LustreError: 280508:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8941.063162] Lustre: Skipped 1 previous similar message [ 8945.120955] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8945.443899] Lustre: server umount lustre-MDT0001 complete [ 8952.616472] Lustre: server umount lustre-OST0000 complete [ 8959.909098] Lustre: server umount lustre-OST0001 complete [ 8968.823934] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 04:32:08 (1787128328) [ 9000.335977] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 9010.834563] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 9047.916410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9048.193844] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9048.219701] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9048.280987] Lustre: lustre-MDT0000: new disk, initializing [ 9048.351269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9048.355765] Lustre: Skipped 1 previous similar message [ 9048.368856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9052.456599] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9062.003436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9062.111099] Lustre: 284835:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9062.162846] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9062.168961] Lustre: Skipped 1 previous similar message [ 9062.196122] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9062.235905] Lustre: lustre-MDT0001: new disk, initializing [ 9062.315423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9062.342043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9062.359961] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9066.577361] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9076.619275] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9081.642557] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9088.775045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9089.092514] Lustre: lustre-OST0000: new disk, initializing [ 9089.097539] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9089.107178] Lustre: 286803:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9089.204449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9095.875439] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9098.324716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9098.330437] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9098.467400] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9104.896958] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9110.567910] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9125.670415] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9132.028184] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9185.777643] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9185.778486] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9185.799463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9185.805719] Lustre: Skipped 1 previous similar message [ 9185.824716] LustreError: Skipped 1 previous similar message [ 9189.340979] Lustre: server umount lustre-OST0000 complete [ 9195.489301] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9195.499083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9199.896731] Lustre: server umount lustre-MDT0000 complete [ 9201.121885] LustreError: 288022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9201.143923] LustreError: 288022:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9203.772680] LustreError: 287658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787128564 with bad export cookie 12811114012279559458 [ 9203.774937] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9203.785468] LustreError: 287658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9204.127860] Lustre: server umount lustre-MDT0001 complete [ 9213.141679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9217.685216] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9222.618401] Lustre: server umount MGS complete [ 9234.971609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9235.394097] LustreError: 290719:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9235.420722] LustreError: 290732:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9235.471575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9239.631526] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9245.680770] LustreError: 290720:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9245.709169] LustreError: 290720:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9248.309861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9253.565768] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9261.178431] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9267.065282] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9273.622591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9273.993441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9274.006455] Lustre: Skipped 1 previous similar message [ 9280.631727] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9288.218364] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9317.357515] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9317.367952] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9317.382734] Lustre: Skipped 1 previous similar message [ 9317.391328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9317.395313] Lustre: Skipped 3 previous similar messages [ 9321.491373] Lustre: server umount lustre-OST0000 complete [ 9325.538976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9325.556987] LustreError: Skipped 1 previous similar message [ 9325.574357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9325.598457] Lustre: Skipped 1 previous similar message [ 9331.652646] Lustre: server umount lustre-MDT0000 complete [ 9334.754413] LustreError: 290715:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9334.782076] LustreError: 290715:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9336.448672] LustreError: 290701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787128697 with bad export cookie 12811114012279561278 [ 9336.450855] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9336.469412] LustreError: 290701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9336.896350] Lustre: server umount lustre-MDT0001 complete [ 9348.299904] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 9350.992034] Key type lgssc unregistered [ 9351.378959] LNet: 294666:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9351.385562] LNetError: 294666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9352.423059] LNet: Removed LNI 192.168.204.142@tcp [ 9353.578631] Key type .llcrypt unregistered [ 9353.587314] Key type ._llcrypt unregistered [ 9397.247543] Key type ._llcrypt registered [ 9397.252111] Key type .llcrypt registered [ 9397.393223] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 9417.200239] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 9418.278736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9418.319454] alg: No test for adler32 (adler32-zlib) [ 9419.477646] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 9419.912882] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 9421.696224] Key type lgssc registered [ 9423.101989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9460.739711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9460.828379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9462.103565] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9462.142910] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9462.255188] Lustre: lustre-MDT0000: new disk, initializing [ 9462.344319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9462.370677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9467.545734] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9479.518319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9479.657896] Lustre: 298625:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9479.752101] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9479.760447] Lustre: Skipped 1 previous similar message [ 9479.829505] Lustre: lustre-MDT0001: new disk, initializing [ 9479.958270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9480.038674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9480.054749] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9484.291333] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9493.449798] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9499.599224] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9507.821911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9508.167172] Lustre: lustre-OST0000: new disk, initializing [ 9508.173102] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9508.193577] Lustre: 300597:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9508.375165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9516.191179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9516.229618] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9516.395437] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9516.481281] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9528.130289] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9535.357817] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9545.800316] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9550.094300] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9555.944290] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 9593.321746] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9593.325497] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9593.338034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9593.341755] LustreError: Skipped 1 previous similar message [ 9593.351305] Lustre: Skipped 1 previous similar message [ 9595.331241] Lustre: server umount lustre-OST0000 complete [ 9603.044636] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9603.056554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9603.066629] Lustre: Skipped 1 previous similar message [ 9603.075585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9605.359809] Lustre: server umount lustre-MDT0000 complete [ 9608.588221] LustreError: 298619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787128969 with bad export cookie 4234530191960305475 [ 9608.603956] LustreError: 298619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9608.606477] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9608.967993] Lustre: server umount lustre-MDT0001 complete [ 9617.023543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9621.751362] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9626.593988] Lustre: server umount MGS complete [ 9639.994245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9640.700217] LustreError: 304386:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9640.787796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9646.050099] LustreError: 304373:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9646.084566] LustreError: 304373:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9646.619907] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9651.171346] LustreError: 304374:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9656.291509] LustreError: 304373:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9658.245892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9658.670435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9665.354608] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9675.016390] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9681.503925] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9688.450785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9688.973234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9696.159194] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9704.625433] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9741.281531] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9741.289442] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9741.303766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9741.329435] Lustre: Skipped 2 previous similar messages [ 9741.348832] Lustre: Skipped 2 previous similar messages [ 9746.404173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9746.418081] Lustre: Skipped 1 previous similar message [ 9747.282905] Lustre: server umount lustre-OST0000 complete [ 9756.129468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9756.148359] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9756.160383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9766.368696] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9766.666739] Lustre: server umount lustre-MDT0000 complete [ 9766.888927] LustreError: 304374:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9766.907276] LustreError: 304374:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9772.017502] LustreError: 304373:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9772.043168] LustreError: 304373:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9772.187684] LustreError: 304353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787129132 with bad export cookie 4234530191960307393 [ 9772.191679] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9772.207202] LustreError: 304353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9772.809955] Lustre: server umount lustre-MDT0001 complete [ 9785.350958] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 9788.608208] Key type lgssc unregistered [ 9788.957898] LNet: 308328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9788.961277] LNetError: 308328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9788.979903] LNet: Removed LNI 192.168.204.142@tcp [ 9790.019639] Key type .llcrypt unregistered [ 9790.023876] Key type ._llcrypt unregistered [ 9815.553789] Key type ._llcrypt registered [ 9815.556987] Key type .llcrypt registered [ 9815.718927] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 9833.906790] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 9835.417070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9835.522535] alg: No test for adler32 (adler32-zlib) [ 9836.903492] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [ 9837.530947] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 9839.394348] Key type lgssc registered [ 9840.632859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9871.804467] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 9890.179220] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9890.225838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9891.516107] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9891.566870] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9891.742622] Lustre: lustre-MDT0000: new disk, initializing [ 9891.894708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9891.946305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9897.416435] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9909.143863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9909.407515] Lustre: lustre-OST0000: new disk, initializing [ 9909.413672] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9909.420104] Lustre: Skipped 1 previous similar message [ 9909.425154] Lustre: 312381:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9909.518375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9915.545097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9915.554660] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9915.589795] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9917.180130] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10793.957233] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10794.216794] Lustre: server umount lustre-MDT0000 complete [10798.035766] LustreError: 312380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787130158 with bad export cookie 15052871998636806348 [10798.039197] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10798.045353] LustreError: 312380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10800.199370] Lustre: server umount lustre-OST0000 complete [10825.171205] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [10836.258661] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [10877.851143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10878.166333] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10878.242972] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10878.363829] Lustre: lustre-MDT0000: new disk, initializing [10878.481854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10878.506883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10883.804524] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10894.215000] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10894.340656] Lustre: 316738:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10894.394478] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10894.397849] Lustre: Skipped 1 previous similar message [10894.473778] Lustre: lustre-MDT0001: new disk, initializing [10894.610264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10894.662955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10894.677645] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10900.615592] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10911.512177] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10917.598947] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10925.486072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10925.818198] Lustre: lustre-OST0000: new disk, initializing [10925.824278] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10925.829870] Lustre: 318706:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10925.957535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10932.380752] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10933.802861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10933.814562] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10933.960200] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10942.067646] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10947.284861] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10947.543220] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10951.938329] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10952.127384] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10954.728613] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10954.732290] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10954.748133] LustreError: Skipped 1 previous similar message [10954.757881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10954.766417] Lustre: Skipped 1 previous similar message [10954.778092] Lustre: Skipped 1 previous similar message [10959.851503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10959.868376] Lustre: Skipped 1 previous similar message [10960.451451] Lustre: server umount lustre-OST0000 complete [10964.977237] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10964.996189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10965.003247] Lustre: Skipped 1 previous similar message [10970.084062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10970.091773] Lustre: Skipped 1 previous similar message [10975.202225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10975.205482] Lustre: Skipped 1 previous similar message [10978.273545] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10978.524555] Lustre: server umount lustre-MDT0000 complete [10980.332600] LustreError: 316752:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10980.367559] LustreError: 316752:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10982.366848] LustreError: 316731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787130343 with bad export cookie 15052871998639747167 [10982.374945] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10982.385617] LustreError: 316731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10982.865739] Lustre: server umount lustre-MDT0001 complete [10987.561412] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [10989.467151] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 05:05:48 (1787130348) [10996.437595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10996.792931] LustreError: 321021:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10996.867323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11001.964980] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11002.339447] LustreError: 321022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11006.458618] LustreError: 321021:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11011.554521] LustreError: 321022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11011.623980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11011.873235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11016.496945] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11027.382254] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11035.085513] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11042.588653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11042.808471] LustreError: 322842:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11043.229828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11051.284975] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11059.977811] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11068.080627] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11068.267939] Lustre: lustre-OST0001: new disk, initializing [11068.272866] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11068.280262] Lustre: 323859:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11068.370273] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11075.223259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11075.226158] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11075.242121] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11075.300666] LustreError: 324214:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11076.326414] LustreError: 324214:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11076.342761] LustreError: 324214:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [11077.413453] LustreError: 324219:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11077.434900] LustreError: 324219:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [11079.472281] LustreError: 324214:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11079.503661] LustreError: 324214:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [11083.632632] LustreError: 324214:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11083.656114] LustreError: 324214:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [11086.418538] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11086.684941] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11091.880397] LustreError: 324219:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11091.905583] LustreError: 324219:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 16 previous similar messages [11092.788417] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8cf244848000.ost_server_uuid 50 [11094.226257] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8cf244848000.ost_server_uuid in FULL state after 0 sec [11099.678678] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8cf244848000.ost_server_uuid 50 [11101.498037] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8cf244848000.ost_server_uuid in FULL state after 0 sec [11103.295810] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11118.573089] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11118.579083] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11118.583306] LustreError: Skipped 1 previous similar message [11118.586924] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11118.597433] Lustre: Skipped 2 previous similar messages [11118.611951] Lustre: Skipped 1 previous similar message [11131.360880] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11131.539323] Lustre: server umount lustre-OST0001 complete [11131.885187] LustreError: 323578:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11131.908222] LustreError: 323578:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11136.995552] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11137.008600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11137.017845] Lustre: Skipped 1 previous similar message [11137.033611] Lustre: Skipped 7 previous similar messages [11143.195516] Lustre: server umount lustre-OST0000 complete [11150.307930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11150.318566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11154.448989] Lustre: server umount lustre-MDT0000 complete [11159.013950] LustreError: 321003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787130519 with bad export cookie 15052871998639748049 [11159.014706] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11159.023297] LustreError: 321003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11159.546445] Lustre: server umount lustre-MDT0001 complete [11170.867850] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [11173.936963] Key type lgssc unregistered [11174.212564] LNet: 326554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11174.217806] LNetError: 326554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11175.272108] LNet: Removed LNI 192.168.204.142@tcp [11176.281932] Key type .llcrypt unregistered [11176.284390] Key type ._llcrypt unregistered [11202.949218] Key type ._llcrypt registered [11202.950686] Key type .llcrypt registered [11203.109079] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [11204.766961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11204.800653] alg: No test for adler32 (adler32-zlib) [11205.995821] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [11206.369723] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [11208.152167] Key type lgssc registered [11209.701583] Lustre: Echo OBD driver; http://www.lustre.org/ [11219.418586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11219.443436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11221.132032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11226.564311] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11230.693986] LustreError: 327977:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11230.712348] LustreError: 327977:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11234.549531] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11235.811435] LustreError: 327978:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11240.931979] LustreError: 327977:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11242.615227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11243.052681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11248.312244] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11257.912155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11258.268166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11264.807474] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11265.460098] LustreError: 329657:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11265.477434] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [11272.334901] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11272.684974] LustreError: 329655:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11272.702142] LustreError: 329655:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11278.767874] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11279.039313] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11284.486948] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [11285.326225] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11293.350463] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11300.526996] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [11300.706550] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11407.333337] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11407.342513] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11407.356274] LustreError: Skipped 1 previous similar message [11407.362461] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11407.375116] Lustre: Skipped 1 previous similar message [11407.416569] Lustre: Skipped 1 previous similar message [11410.482810] Lustre: server umount lustre-OST0001 complete [11412.462386] LustreError: 330957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11412.493893] LustreError: 330957:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11419.108990] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11419.119851] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11419.148855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11421.156299] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11421.171522] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11421.194402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11424.208361] Lustre: server umount lustre-OST0000 complete [11432.419432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11432.431521] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11432.448684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11432.461731] Lustre: Skipped 2 previous similar messages [11438.050460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11438.055374] Lustre: Skipped 2 previous similar messages [11443.173843] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11443.428656] Lustre: server umount lustre-MDT0000 complete [11448.287403] LustreError: 327957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787130808 with bad export cookie 400909481103128473 [11448.299591] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11448.312947] LustreError: 327957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11448.313218] LustreError: 327977:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11448.361270] LustreError: 327977:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11448.959841] Lustre: server umount lustre-MDT0001 complete [11462.735224] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [11465.604960] Key type lgssc unregistered [11465.915177] LNet: 332929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11465.924781] LNetError: 332929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11465.955640] LNet: Removed LNI 192.168.204.142@tcp [11466.751120] Key type .llcrypt unregistered [11466.757078] Key type ._llcrypt unregistered [11516.834412] Key type ._llcrypt registered [11516.836675] Key type .llcrypt registered [11516.957426] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [11518.153069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11518.350093] alg: No test for adler32 (adler32-zlib) [11519.412960] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [11519.919633] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [11521.664256] Key type lgssc registered [11522.916852] Lustre: Echo OBD driver; http://www.lustre.org/ [11533.670351] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11533.682531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11535.108635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11539.890593] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11544.557130] LustreError: 335220:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11544.590412] LustreError: 335220:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11547.681314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11548.046737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11552.458610] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11555.588545] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11562.558605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11563.076635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11570.013817] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11572.329182] LustreError: 336711:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11577.843581] LustreError: 336712:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11579.452390] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11579.703754] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11580.741427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [11580.741740] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [11587.233409] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11599.596574] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11607.455613] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11621.863792] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11621.868296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11621.888705] Lustre: Skipped 3 previous similar messages [11621.895420] Lustre: Skipped 2 previous similar messages [11626.594111] Lustre: server umount lustre-MDT0000 complete [11626.980578] LustreError: 335221:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11627.003236] LustreError: 335221:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11630.372929] LustreError: 335202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787130991 with bad export cookie 3198162941717955506 [11630.377295] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11630.386609] LustreError: 335202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11630.764875] Lustre: server umount lustre-MDT0001 complete [11645.277269] Lustre: server umount lustre-OST0000 complete [11647.456130] Lustre: 334482:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787130992/real 1787130992] req@ffff8c528eff2a00 x1873942549694208/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787131008 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11647.479223] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11649.136482] Lustre: server umount lustre-OST0001 complete [11666.495718] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [11676.630891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11677.081116] LustreError: 340484:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11677.163789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11681.832455] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11686.375439] LustreError: 340485:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11686.384774] LustreError: 340485:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11689.820677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11690.265741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11694.436176] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11697.384558] Lustre: 341625:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11705.102029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11712.141496] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11714.807490] LustreError: 342457:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11722.227932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [11722.567325] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11722.787707] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11722.792665] Lustre: Skipped 1 previous similar message [11727.867555] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [11729.726278] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11737.513572] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11741.048617] Lustre: 343495:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11746.786305] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11746.790657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11746.811315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11748.322474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11750.726096] Lustre: server umount lustre-MDT0000 complete [11753.450050] LustreError: 340480:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11753.469486] LustreError: 340480:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [11755.101938] LustreError: 340464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787131115 with bad export cookie 3198162941717957207 [11755.104826] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11755.121735] LustreError: 340464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11755.528569] Lustre: server umount lustre-MDT0001 complete [11770.120584] Lustre: server umount lustre-OST0000 complete [11784.176834] Lustre: server umount lustre-OST0001 complete [11788.580609] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [11790.280494] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [11792.592297] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 05:19:11 (1787131151) [11816.257569] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [11824.795507] loop0: detected capacity change from 0 to 1099511627776 [11825.453603] SGI XFS with ACLs, security attributes, quota, no debug enabled [11825.464268] *** VALIDATE xfs *** [11825.477160] XFS (loop0): Mounting V5 Filesystem [11825.501902] XFS (loop0): Ending clean mount [11826.657889] loop1: detected capacity change from 0 to 18691697672192 [11940.204560] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [11953.099769] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [11989.712044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11989.941575] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11989.966600] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11990.067890] Lustre: lustre-MDT0000: new disk, initializing [11990.158630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11990.183957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11994.329767] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12005.040289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12005.151704] Lustre: 348631:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12005.229420] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12005.232995] Lustre: Skipped 1 previous similar message [12005.323263] Lustre: lustre-MDT0001: new disk, initializing [12005.470304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12005.479518] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12009.704680] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12019.135327] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12026.129313] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12034.624543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12034.857657] Lustre: lustre-OST0000: new disk, initializing [12034.861335] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12034.866591] Lustre: 350601:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12036.484632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12036.500614] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12036.683650] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12042.060741] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12053.658249] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12058.168071] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12058.381423] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12063.080994] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12063.298110] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12067.304285] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12067.318364] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12067.327383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12067.345776] Lustre: Skipped 4 previous similar messages [12067.350637] Lustre: Skipped 3 previous similar messages [12071.847500] Lustre: server umount lustre-OST0000 complete [12077.033171] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12077.065735] LustreError: Skipped 1 previous similar message [12077.070692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12077.105148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12077.113338] Lustre: Skipped 1 previous similar message [12082.578497] Lustre: server umount lustre-MDT0000 complete [12082.661927] LustreError: 348644:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12082.677780] LustreError: 348644:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12087.052118] LustreError: 348625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787131447 with bad export cookie 3198162941717958754 [12087.066777] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12087.711486] Lustre: server umount lustre-MDT0001 complete [12097.528585] XFS (loop0): Unmounting Filesystem [12102.411529] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 05:24:21 (1787131461) [12110.107903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12110.739614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12110.752343] Lustre: Skipped 2 previous similar messages [12115.322076] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12124.213636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12128.799814] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12135.755701] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12141.017835] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12146.874412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12152.719721] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12159.413506] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12170.721171] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12170.728728] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12170.736915] Lustre: Skipped 1 previous similar message [12170.742936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12170.753676] Lustre: Skipped 1 previous similar message [12174.525934] Lustre: server umount lustre-OST0000 complete [12180.964462] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12180.972605] LustreError: Skipped 1 previous similar message [12187.107969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12187.122764] Lustre: Skipped 6 previous similar messages [12193.248715] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12193.364521] Lustre: server umount lustre-MDT0000 complete [12198.195046] LustreError: 353139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787131558 with bad export cookie 3198162941717959636 [12198.204186] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12198.215573] LustreError: 353139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [12198.794941] Lustre: server umount lustre-MDT0001 complete [12216.208891] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12219.566088] Key type lgssc unregistered [12219.846835] LNet: 356815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12219.853115] LNetError: 356815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12219.866365] LNet: Removed LNI 192.168.204.142@tcp [12220.820173] Key type .llcrypt unregistered [12220.822066] Key type ._llcrypt unregistered [12253.584118] Key type ._llcrypt registered [12253.592307] Key type .llcrypt registered [12253.702775] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [12255.093116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12255.192725] alg: No test for adler32 (adler32-zlib) [12256.272539] Lustre: Lustre: Build Version: 2.17.57_3_g9f8445d [12256.468103] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [12258.136201] Key type lgssc registered [12259.428482] Lustre: Echo OBD driver; http://www.lustre.org/ [12267.097900] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12267.121766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12268.382203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12272.660425] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12278.755494] LustreError: 358382:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12278.771106] LustreError: 358382:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12280.643184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12280.977272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12285.218200] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12291.536658] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12296.363415] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12302.788364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12303.156725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12310.954612] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12319.139973] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12328.958025] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12328.997331] Lustre: Skipped 1 previous similar message [12328.999500] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12329.028034] Lustre: Skipped 1 previous similar message [12332.292769] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.42@tcp (stopping) [12333.962134] Lustre: server umount lustre-MDT0001 complete [12334.062044] LustreError: 359109:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12334.083036] LustreError: 359109:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12337.353186] LustreError: 358377:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12342.474711] LustreError: 359971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12342.511663] LustreError: 359971:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12344.109316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12344.191561] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12344.196210] Lustre: Found index 1 for lustre-MDT0001, updating log [12344.244659] Lustre: 358372:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12344.627085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12349.045319] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12349.932360] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12353.017524] LustreError: 361982:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12353.029229] LustreError: 361982:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12353.038051] LustreError: 358374:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12353.096917] LustreError: 361985:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.142@tcp: cfg command failed: rc = -17 [12353.111554] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [12353.111554] [12655.080687] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12655.108034] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12655.127670] Lustre: Skipped 1 previous similar message [12659.949272] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.42@tcp (stopping) [12661.233278] Lustre: server umount lustre-MDT0001 complete [12665.031527] LustreError: 361891:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12665.048972] LustreError: 361891:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12673.965439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12674.028137] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12674.031361] Lustre: Found index 1 for lustre-MDT0001, updating log [12674.060761] Lustre: 358372:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12674.435091] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12679.681662] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12680.358823] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12680.755401] LustreError: 370160:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12680.772374] LustreError: 370160:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12680.797698] LustreError: 358374:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12680.810378] LustreError: 358374:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12680.962254] LustreError: 370169:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [12680.978154] LustreError: 370169:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [12680.993317] LustreError: 370169:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.142@tcp: cfg command failed: rc = -17 [12681.011095] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.204.142@tcp [12681.011095] [12991.985803] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12991.988020] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12992.029032] Lustre: Skipped 1 previous similar message [12992.033529] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12992.033537] Lustre: Skipped 2 previous similar messages [13002.211399] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13002.219050] Lustre: Skipped 5 previous similar messages [13004.256595] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13004.581908] Lustre: server umount lustre-MDT0001 complete [13007.334333] LustreError: 358381:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13007.346541] LustreError: 358381:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [13015.898311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13016.080635] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13016.089844] Lustre: Found index 1 for lustre-MDT0001, updating log [13016.196128] Lustre: 358370:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13016.796599] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13022.203537] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13023.311889] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [13025.267505] LustreError: 378404:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13025.278400] LustreError: 378404:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13025.288424] LustreError: 358374:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13025.293720] LustreError: 358374:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [13025.362025] LustreError: 378410:0:(lod_lov.c:210:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [13025.373336] LustreError: 378410:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.142@tcp: cfg command failed: rc = -114 [13025.383671] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [13025.383671] [13349.348961] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13349.368311] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13349.387202] Lustre: Skipped 1 previous similar message [13349.402414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13349.423113] Lustre: Skipped 2 previous similar messages [13353.233130] Lustre: server umount lustre-MDT0000 complete [13354.979386] LustreError: 358376:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13354.993585] LustreError: 358376:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [13357.963685] LustreError: 358363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787132718 with bad export cookie 16897173830321538276 [13357.984085] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13357.985764] LustreError: 358363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13360.097761] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13360.112905] Lustre: Skipped 2 previous similar messages [13372.384169] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13372.854532] Lustre: server umount lustre-MDT0001 complete [13377.437984] Lustre: server umount lustre-OST0000 complete [13388.606460] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [13402.862855] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [13451.655871] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 05:46:50 (1787132810) [13459.335168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13459.711976] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13459.760882] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13459.845145] Lustre: lustre-MDT0000: new disk, initializing [13459.922635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13459.949606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13464.952976] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [13477.363632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13477.526117] Lustre: 388416:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13477.588187] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13477.591711] Lustre: Skipped 1 previous similar message [13477.657615] Lustre: lustre-MDT0001: new disk, initializing [13477.740056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13477.771444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13477.783970] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13482.170306] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [13493.593697] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13501.383595] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13509.449857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13509.768405] Lustre: lustre-OST0000: new disk, initializing [13509.772476] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13509.778653] Lustre: 390385:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13509.886870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13517.501329] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [13519.930867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13519.940310] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13520.058816] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13529.298809] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13539.307172] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13539.319502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13539.336726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13539.349223] Lustre: Skipped 6 previous similar messages [13540.844693] LustreError: 388421:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13540.877363] LustreError: 388421:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13541.232415] Lustre: server umount lustre-MDT0000 complete [13546.203267] LustreError: 388408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787132906 with bad export cookie 16897173830321540229 [13546.212262] LustreError: 389221:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [13546.215626] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13546.225684] LustreError: 388408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13546.786083] Lustre: server umount lustre-MDT0001 complete [13561.882502] Lustre: server umount lustre-OST0000 complete [13601.662761] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [13615.122214] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [13650.919691] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13339 sec ========= 05:50:10 (1787133010) [13652.635727] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 05:50:12 (1787133012) === [13656.641729] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 05:50:16 (1787133016) === [13709.212388] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [13713.112793] Key type lgssc unregistered [13713.582440] LNet: 396757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13713.592622] LNetError: 396757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13713.622399] LNet: Removed LNI 192.168.204.142@tcp [13715.058119] Key type .llcrypt unregistered [13715.060350] Key type ._llcrypt unregistered