[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 488549060 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002377] x2apic enabled [ 0.003021] Switched APIC routing to physical x2apic. [ 0.004018] kvm-guest: setup PV IPIs [ 0.006746] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007035] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008020] pid_max: default: 32768 minimum: 301 [ 0.010020] LSM: Security Framework initializing [ 0.011071] Yama: becoming mindful. [ 0.012064] SELinux: Initializing. [ 0.014080] *** VALIDATE selinux *** [ 0.023895] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028836] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029279] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030175] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032113] *** VALIDATE tmpfs *** [ 0.034220] *** VALIDATE proc *** [ 0.035250] *** VALIDATE cgroup *** [ 0.036021] *** VALIDATE cgroup2 *** [ 0.037278] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038145] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041032] Spectre V2 : User space: Vulnerable [ 0.042011] Speculative Store Bypass: Vulnerable [ 0.045225] debug: unmapping init [mem 0xffffffffb4659000-0xffffffffb4660fff] [ 0.047169] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048644] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049023] ... version: 2 [ 0.049959] ... bit width: 48 [ 0.050016] ... generic registers: 4 [ 0.050994] ... value mask: 0000ffffffffffff [ 0.051036] ... max period: 00007fffffffffff [ 0.052018] ... fixed-purpose events: 3 [ 0.053017] ... event mask: 000000070000000f [ 0.054363] rcu: Hierarchical SRCU implementation. [ 0.056715] smp: Bringing up secondary CPUs ... [ 0.057726] x86: Booting SMP configuration: [ 0.058033] .... node #0, CPUs: #1 #2 #3 [ 0.062011] smp: Brought up 1 node, 4 CPUs [ 0.064019] smpboot: Max logical packages: 1 [ 0.065013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.123350] node 0 deferred pages initialised in 55ms [ 0.126178] devtmpfs: initialized [ 0.127287] x86/mm: Memory block size: 128MB [ 0.129630] gcov: version magic: 0x41383552 [ 0.133280] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.134084] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.137343] pinctrl core: initialized pinctrl subsystem [ 0.139224] [ 0.139838] ************************************************************* [ 0.142014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.144011] ** ** [ 0.146012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.149014] ** ** [ 0.151011] ** This means that this kernel is built to expose internal ** [ 0.154012] ** IOMMU data structures, which may compromise security on ** [ 0.156007] ** your system. ** [ 0.159012] ** ** [ 0.161013] ** If you see this message and you are not debugging the ** [ 0.163015] ** kernel, report this immediately to your vendor! ** [ 0.166012] ** ** [ 0.168011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171016] ************************************************************* [ 0.174758] NET: Registered protocol family 16 [ 0.176525] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.181071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.184076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.188121] cpuidle: using governor menu [ 0.189850] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.190454] PCI: Using configuration type 1 for base access [ 0.191135] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.201143] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.203021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.207162] cryptd: max_cpu_qlen set to 1000 [ 0.212208] ACPI: Added _OSI(Module Device) [ 0.213014] ACPI: Added _OSI(Processor Device) [ 0.215016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.217015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.222067] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.228478] ACPI: Interpreter enabled [ 0.229045] ACPI: PM: (supports S0 S3 S4 S5) [ 0.230013] ACPI: Using IOAPIC for interrupt routing [ 0.232131] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.236424] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.245500] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.248050] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.251021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.254060] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.259424] acpiphp: Slot [2] registered [ 0.260107] acpiphp: Slot [5] registered [ 0.262127] acpiphp: Slot [6] registered [ 0.264123] acpiphp: Slot [7] registered [ 0.265131] acpiphp: Slot [8] registered [ 0.267216] acpiphp: Slot [9] registered [ 0.269178] acpiphp: Slot [10] registered [ 0.270148] acpiphp: Slot [3] registered [ 0.272095] acpiphp: Slot [4] registered [ 0.273068] acpiphp: Slot [11] registered [ 0.274085] acpiphp: Slot [12] registered [ 0.276104] acpiphp: Slot [13] registered [ 0.278111] acpiphp: Slot [14] registered [ 0.279108] acpiphp: Slot [15] registered [ 0.280101] acpiphp: Slot [16] registered [ 0.282107] acpiphp: Slot [17] registered [ 0.283084] acpiphp: Slot [18] registered [ 0.285092] acpiphp: Slot [19] registered [ 0.286111] acpiphp: Slot [20] registered [ 0.288105] acpiphp: Slot [21] registered [ 0.290120] acpiphp: Slot [22] registered [ 0.291126] acpiphp: Slot [23] registered [ 0.293105] acpiphp: Slot [24] registered [ 0.294101] acpiphp: Slot [25] registered [ 0.296117] acpiphp: Slot [26] registered [ 0.297118] acpiphp: Slot [27] registered [ 0.298090] acpiphp: Slot [28] registered [ 0.300110] acpiphp: Slot [29] registered [ 0.301129] acpiphp: Slot [30] registered [ 0.303132] acpiphp: Slot [31] registered [ 0.305043] PCI host bridge to bus 0000:00 [ 0.307021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.308020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.311022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.312019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.314026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.317025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.318244] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.322204] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.324255] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.337015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.343043] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.345029] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.348020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.351019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.354456] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.356872] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.360052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.363725] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.367017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.382016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.387017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.394042] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.405016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.413016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.431018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.441208] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.451017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.459018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.475025] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.484381] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.493016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.500022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.520015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.533057] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.540021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.547016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.578016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.593725] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.600014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.605015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.620024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.630317] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.637015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.644015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.662017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.674981] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.677395] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.680338] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.682356] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.684219] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.688033] iommu: Default domain type: Passthrough [ 0.689311] SCSI subsystem initialized [ 0.691130] ACPI: bus type USB registered [ 0.692126] usbcore: registered new interface driver usbfs [ 0.694095] usbcore: registered new interface driver hub [ 0.696083] usbcore: registered new device driver usb [ 0.698144] pps_core: LinuxPPS API ver. 1 registered [ 0.700008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.704069] PTP clock support registered [ 0.706070] EDAC MC: Ver: 3.0.0 [ 0.708117] PCI: Using ACPI for IRQ routing [ 0.710973] NetLabel: Initializing [ 0.712015] NetLabel: domain hash size = 128 [ 0.713008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.715067] NetLabel: unlabeled traffic allowed by default [ 0.717080] vgaarb: loaded [ 0.718246] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.720015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.726017] clocksource: Switched to clocksource kvm-clock [ 0.838553] VFS: Disk quotas dquot_6.6.0 [ 0.839937] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.842525] *** VALIDATE ramfs *** [ 0.843740] *** VALIDATE hugetlbfs *** [ 0.845965] pnp: PnP ACPI init [ 0.848294] pnp: PnP ACPI: found 6 devices [ 0.866031] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.868585] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.870530] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.872142] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.874095] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.875965] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.878127] NET: Registered protocol family 2 [ 0.880078] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.884531] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.887865] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.892356] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.895484] TCP: Hash tables configured (established 65536 bind 65536) [ 0.897700] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.900091] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.902259] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.904472] NET: Registered protocol family 1 [ 0.906676] RPC: Registered named UNIX socket transport module. [ 0.908181] RPC: Registered udp transport module. [ 0.909454] RPC: Registered tcp transport module. [ 0.910717] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.912413] NET: Registered protocol family 44 [ 0.913627] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.915167] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.916753] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.918485] PCI: CLS 0 bytes, default 64 [ 0.920647] Unpacking initramfs... [ 2.352262] debug: unmapping init [mem 0xffff97b63cc54000-0xffff97b63ffbffff] [ 2.356241] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.358205] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.360672] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.869515] Initialise system trusted keyrings [ 2.871494] Key type blacklist registered [ 2.874315] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.883523] zbud: loaded [ 2.887259] *** VALIDATE nfs *** [ 2.888816] *** VALIDATE nfs4 *** [ 2.890526] pstore: using deflate compression [ 2.894468] Platform Keyring initialized [ 2.993392] NET: Registered protocol family 38 [ 2.995588] Key type asymmetric registered [ 2.997203] Asymmetric key parser 'x509' registered [ 2.999380] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.002631] io scheduler mq-deadline registered [ 3.004311] io scheduler kyber registered [ 3.006048] io scheduler bfq registered [ 3.007897] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.010572] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.013237] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.015884] ACPI: Power Button [PWRF] [ 3.022587] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.030206] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.053770] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.064068] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.091397] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.122499] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.152162] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.157913] Non-volatile memory driver v1.3 [ 3.159834] Linux agpgart interface v0.103 [ 3.191101] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.194234] vda: detected capacity change from 0 to 74698752 [ 3.210556] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.213192] vdb: detected capacity change from 0 to 1073741824 [ 3.234558] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.237719] vdc: detected capacity change from 0 to 2621440000 [ 3.259975] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.263189] vdd: detected capacity change from 0 to 2621440000 [ 3.284699] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.287110] vde: detected capacity change from 0 to 4294967296 [ 3.303247] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.306764] vdf: detected capacity change from 0 to 4294967296 [ 3.321261] libphy: Fixed MDIO Bus: probed [ 3.331421] usbcore: registered new interface driver usbserial_generic [ 3.334389] usbserial: USB Serial support registered for generic [ 3.336903] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.341943] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.344513] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.347458] mousedev: PS/2 mouse device common for all mice [ 3.350334] rtc_cmos 00:05: RTC can wake from S4 [ 3.352807] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.353491] rtc_cmos 00:05: registered as rtc0 [ 3.357787] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.359787] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.365692] intel_pstate: CPU model not supported [ 3.367214] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.373699] hid: raw HID events driver (C) Jiri Kosina [ 3.376740] usbcore: registered new interface driver usbhid [ 3.379369] usbhid: USB HID core driver [ 3.381592] drop_monitor: Initializing network drop monitor service [ 3.384738] Initializing XFRM netlink socket [ 3.387340] NET: Registered protocol family 10 [ 3.390267] Segment Routing with IPv6 [ 3.391968] NET: Registered protocol family 17 [ 3.394419] mpls_gso: MPLS GSO support [ 3.402851] RAS: Correctable Errors collector initialized. [ 3.405265] AVX version of gcm_enc/dec engaged. [ 3.407070] AES CTR mode by8 optimization enabled [ 3.484668] sched_clock: Marking stable (3484642267, 0)->(4429159209, -944516942) [ 3.488200] registered taskstats version 1 [ 3.490195] Loading compiled-in X.509 certificates [ 3.492472] zswap: loaded using pool lzo/zbud [ 3.517287] Key type big_key registered [ 3.536595] Key type encrypted registered [ 3.538538] ima: No TPM chip found, activating TPM-bypass! [ 3.540409] ima: Allocated hash algorithm: sha1 [ 3.542128] ima: No architecture policies found [ 3.544203] evm: Initialising EVM extended attributes: [ 3.546798] evm: security.selinux [ 3.548307] evm: security.ima [ 3.549583] evm: security.capability [ 3.550956] evm: HMAC attrs: 0x1 [ 3.553774] rtc_cmos 00:05: setting system clock to 2026-09-03 13:22:37 UTC (1788441757) [ 3.559881] debug: unmapping init [mem 0xffffffffb5603000-0xffffffffb57fffff] [ 3.563689] debug: unmapping init [mem 0xffffffffb4382000-0xffffffffb4658fff] [ 3.570123] Write protecting the kernel read-only data: 28672k [ 3.573907] debug: unmapping init [mem 0xffffffffb2a03000-0xffffffffb2bfffff] [ 3.576891] debug: unmapping init [mem 0xffffffffb3314000-0xffffffffb33fffff] [ 3.616831] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.626636] systemd[1]: Detected virtualization kvm. [ 3.628766] systemd[1]: Detected architecture x86-64. [ 3.630731] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.658652] systemd[1]: No hostname configured. [ 3.660763] systemd[1]: Set hostname to . [ 3.663257] random: systemd: uninitialized urandom read (16 bytes read) [ 3.666118] systemd[1]: Initializing machine ID from random generator. [ 3.725557] random: ln: uninitialized urandom read (6 bytes read) [ 3.823673] random: systemd: uninitialized urandom read (16 bytes read) [ 3.826865] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.833677] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.840477] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.558286] device-mapper: uevent: version 1.0.3 [ 4.561564] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.306513] random: fast init done [ 5.344821] virtio_net virtio0 ens2: renamed from eth0 [ 5.422718] scsi host0: ata_piix [ 5.430245] scsi host1: ata_piix [ 5.431488] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.433248] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.094603] dracut-initqueue[582]: RTNETLINK answers: File exists [ 10.056133] random: crng init done [ 10.057566] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.566433] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.695300] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.969443] SELinux: Disabled at runtime. [ 12.034979] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.044388] systemd[1]: Detected virtualization kvm. [ 12.046721] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.610174] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.613898] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.621486] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.625930] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.628696] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.635339] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.652459] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... Mounting Huge Pages File System... Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ 12.721406] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.032467] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.310357] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.322787] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.476842] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.494553] EDAC sbridge: Ver: 1.1.2 [ 15.346147] Key type dns_resolver registered [ 15.655221] NFS: Registering the id_resolver key type [ 15.658221] Key type id_resolver registered [ 15.659295] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg401-server login: [ 39.522650] hrtimer: interrupt took 6101338 ns [ 50.795270] libcfs: loading out-of-tree module taints kernel. [ 50.875851] Key type ._llcrypt registered [ 50.879604] Key type .llcrypt registered [ 51.019515] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [ 73.311076] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 75.348476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 75.368060] alg: No test for adler32 (adler32-zlib) [ 77.234941] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 78.310442] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 80.128237] Key type lgssc registered [ 82.556890] Lustre: Echo OBD driver; http://www.lustre.org/ [ 100.338765] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 144.762531] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 160.472178] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 160.559077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 161.942311] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 162.032515] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 162.198968] Lustre: lustre-MDT0000: new disk, initializing [ 162.346161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 162.390046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 167.583888] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 183.035338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 183.147723] Lustre: 6510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 183.177687] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 183.182518] Lustre: Skipped 1 previous similar message [ 183.301597] Lustre: lustre-MDT0001: new disk, initializing [ 183.371528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 183.409032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 183.423878] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 187.219243] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 191.796665] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 203.083106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 203.394331] Lustre: lustre-OST0000: new disk, initializing [ 203.402434] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 203.412982] Lustre: 8450:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 203.496118] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 209.497098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 209.511287] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 209.657672] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 212.221674] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 229.185792] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 229.326394] Lustre: lustre-OST0001: new disk, initializing [ 229.330167] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 229.334907] Lustre: 9524:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 229.406235] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 236.031234] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 238.117686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 238.124250] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 238.218336] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 249.080898] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 256.212392] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 263.383639] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing check_logdir /tmp/testlogs/ [ 268.885159] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing yml_node [ 273.766427] Lustre: DEBUG MARKER: Client: 2.17.56.51 [ 276.126428] Lustre: DEBUG MARKER: MDS: 2.17.56.51 [ 279.199890] Lustre: DEBUG MARKER: OSS: 2.17.56.51 [ 281.012518] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 09:27:13 EDT 2026 [ 302.748976] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 305.256820] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 325.099342] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 325.101580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 325.105193] Lustre: Skipped 3 previous similar messages [ 325.109169] Lustre: Skipped 2 previous similar messages [ 330.210183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 330.225425] Lustre: Skipped 4 previous similar messages [ 335.329948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 335.335661] Lustre: Skipped 3 previous similar messages [ 336.352154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 336.579177] Lustre: server umount lustre-MDT0000 complete [ 341.468421] LustreError: 9523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442095 with bad export cookie 8105223507319914488 [ 341.469161] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 341.491373] LustreError: 9523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 345.569096] LustreError: 6519:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 345.570259] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 345.571549] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 345.586967] LustreError: 6519:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 345.621492] Lustre: Skipped 1 previous similar message [ 346.595421] LustreError: 6518:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 346.617233] LustreError: 6518:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 348.017586] Lustre: server umount lustre-MDT0001 complete [ 352.597498] Lustre: server umount lustre-OST0000 complete [ 357.035476] Lustre: server umount lustre-OST0001 complete [ 364.901251] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [ 375.536721] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 416.146034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 416.477057] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 416.530410] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 416.661491] Lustre: lustre-MDT0000: new disk, initializing [ 416.768310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 416.794329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 421.595973] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 434.046758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 434.187643] Lustre: 15078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 434.254800] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 434.259548] Lustre: Skipped 1 previous similar message [ 434.378947] Lustre: lustre-MDT0001: new disk, initializing [ 434.518098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 434.598869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 434.623418] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 439.610611] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 449.454694] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 456.588853] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 464.966246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 465.313123] Lustre: lustre-OST0000: new disk, initializing [ 465.319839] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 465.333488] Lustre: 17050:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 465.432468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 466.864591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 466.873726] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 467.076123] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 472.762994] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 483.786435] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 489.183367] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 489.445231] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 494.650670] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 494.991690] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 497.636983] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 497.639453] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 497.653185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 497.653194] Lustre: Skipped 3 previous similar messages [ 503.308726] Lustre: server umount lustre-OST0000 complete [ 507.875431] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 507.887130] Lustre: Skipped 1 previous similar message [ 507.890601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 507.895191] Lustre: Skipped 4 previous similar messages [ 513.610725] Lustre: server umount lustre-MDT0000 complete [ 518.126807] LustreError: 15086:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 518.150025] LustreError: 15086:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 518.302378] LustreError: 15072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442272 with bad export cookie 8105223507319916014 [ 518.316804] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 518.335207] LustreError: 15072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 518.665307] Lustre: server umount lustre-MDT0001 complete [ 524.670683] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 09:31:16 (1788442276) [ 532.600710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 533.115814] LustreError: 19313:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 533.222204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 538.608195] LustreError: 19314:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 538.636739] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 548.838851] LustreError: 19313:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 548.877978] LustreError: 19313:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 549.092952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 555.143751] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 563.819383] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 570.471488] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 578.458053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 586.062577] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 593.883967] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 609.182864] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 618.983209] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 618.988506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 618.989589] Lustre: Skipped 2 previous similar messages [ 618.997802] Lustre: Skipped 4 previous similar messages [ 623.107546] Lustre: server umount lustre-MDT0000 complete [ 624.097851] LustreError: 19314:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 624.134108] LustreError: 19314:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 627.277174] LustreError: 19293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442381 with bad export cookie 8105223507319916896 [ 627.282434] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 627.289606] LustreError: 19293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 627.571839] Lustre: server umount lustre-MDT0001 complete [ 642.761692] Lustre: server umount lustre-OST0000 complete [ 667.847807] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 679.254193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 679.644621] LustreError: 23720:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 679.795539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 679.801932] Lustre: Skipped 2 previous similar messages [ 684.659155] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 694.413603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 699.308952] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 702.304256] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 708.817941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 715.850323] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 725.239494] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 725.548579] Lustre: lustre-OST0001: new disk, initializing [ 725.561133] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 725.589573] Lustre: 26034:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 726.959682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 726.968728] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 727.161807] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 733.377873] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 743.720327] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 747.629352] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 761.315065] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 761.325381] LustreError: Skipped 1 previous similar message [ 761.329776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 761.346067] Lustre: Skipped 1 previous similar message [ 761.351103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 767.035510] Lustre: server umount lustre-MDT0000 complete [ 767.979874] LustreError: 26088:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 768.005498] LustreError: 26088:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 770.731856] LustreError: 24462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442524 with bad export cookie 8105223507319917750 [ 770.739422] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 770.742544] LustreError: 24462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 771.037907] Lustre: server umount lustre-MDT0001 complete [ 785.491771] Lustre: server umount lustre-OST0000 complete [ 789.474627] Lustre: 3646:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442527/real 1788442527] req@ffff97b689aea300 x1875317182689280/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788442543 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 789.525635] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 789.549850] Lustre: Skipped 3 previous similar messages [ 789.989358] Lustre: server umount lustre-OST0001 complete [ 809.045613] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 820.365883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 820.968215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 820.981275] Lustre: Skipped 3 previous similar messages [ 825.797854] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 834.805619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 840.568747] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 843.742389] Lustre: 30203:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 851.186489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 858.733913] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 868.187136] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 874.833697] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 883.097385] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 887.156315] Lustre: 32080:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 894.451663] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 894.458540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 894.483846] Lustre: Skipped 2 previous similar messages [ 894.509475] Lustre: Skipped 7 previous similar messages [ 900.305068] Lustre: server umount lustre-MDT0000 complete [ 904.588829] LustreError: 31392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442658 with bad export cookie 8105223507319919038 [ 904.601950] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 904.611159] LustreError: 31392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 904.676419] LustreError: 29051:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 904.702863] LustreError: 29051:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 905.092567] Lustre: server umount lustre-MDT0001 complete [ 911.339271] Lustre: server umount lustre-OST0000 complete [ 915.496384] Lustre: server umount lustre-OST0001 complete [ 925.044949] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 09:37:57 (1788442677) [ 956.876912] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 968.837121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 974.787485] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 984.443140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 989.299302] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 992.019604] Lustre: 36034:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 999.238360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1006.428285] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1016.224819] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1024.284595] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1033.542239] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1038.343767] Lustre: 37908:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1046.497777] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1046.517080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1046.532766] Lustre: Skipped 3 previous similar messages [ 1046.542610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1046.559029] Lustre: Skipped 6 previous similar messages [ 1051.124996] Lustre: server umount lustre-MDT0000 complete [ 1055.774759] LustreError: 34865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442809 with bad export cookie 8105223507319920151 [ 1055.794330] LustreError: 34865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1055.795605] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1056.316206] Lustre: server umount lustre-MDT0001 complete [ 1072.178506] Lustre: server umount lustre-OST0000 complete [ 1073.954412] Lustre: 3646:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442811/real 1788442811] req@ffff97b583561f80 x1875317182790144/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788442827 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1076.194397] Lustre: 3644:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442814/real 1788442814] req@ffff97b6b3397800 x1875317182790400/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788442830 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1076.852836] Lustre: server umount lustre-OST0001 complete [ 1086.910628] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 09:40:38 (1788442838) [ 1107.154729] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 1119.271200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1119.988817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1120.007433] Lustre: Skipped 7 previous similar messages [ 1125.129231] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1135.450184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1140.786957] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1144.048143] Lustre: 41202:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1152.855666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1159.837434] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1160.699613] LustreError: 41555:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1160.746385] LustreError: 41555:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 1170.045650] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1178.514384] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1189.162695] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1193.683207] Lustre: 43080:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1194.715962] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1194.718756] Lustre: Skipped 1 previous similar message [ 1201.143875] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1201.159621] Lustre: Skipped 5 previous similar messages [ 1205.190045] Lustre: server umount lustre-MDT0000 complete [ 1209.183383] LustreError: 40035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442963 with bad export cookie 8105223507319921257 [ 1209.195115] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1209.195800] LustreError: 40035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1215.753362] Lustre: server umount lustre-MDT0001 complete [ 1226.052773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1240.122885] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1244.588483] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1250.159320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1255.942882] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1255.948997] Lustre: Skipped 1 previous similar message [ 1256.544861] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1265.532039] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1267.332188] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1273.652038] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1275.485493] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1281.509880] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1287.876891] Lustre: server umount lustre-MDT0000 complete [ 1291.912972] LustreError: 40035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443045 with bad export cookie 8105223507319922356 [ 1291.922227] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1291.927234] LustreError: 40035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1292.273307] Lustre: server umount lustre-MDT0001 complete [ 1306.270258] Lustre: server umount lustre-OST0000 complete [ 1320.863593] Lustre: server umount lustre-OST0001 complete [ 1331.646441] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 09:44:43 (1788443083) [ 1351.921744] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 1364.020451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1369.681484] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1380.662730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1387.520889] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1391.116947] Lustre: 48846:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1399.535481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1407.453862] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1419.698804] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1427.524290] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1437.840907] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1444.446294] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1461.233882] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1461.252270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1461.252939] Lustre: Skipped 9 previous similar messages [ 1461.265649] Lustre: Skipped 25 previous similar messages [ 1467.417767] Lustre: server umount lustre-MDT0000 complete [ 1471.592824] LustreError: 47674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443225 with bad export cookie 8105223507319923175 [ 1471.611350] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1471.622681] LustreError: 47674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1472.033671] Lustre: server umount lustre-MDT0001 complete [ 1487.284330] Lustre: server umount lustre-OST0000 complete [ 1500.466822] Lustre: server umount lustre-OST0001 complete [ 1514.407975] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 09:47:45 (1788443265) [ 1517.574570] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1519.517682] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 09:47:51 (1788443271) [ 1530.736974] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [ 1542.008952] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 1571.864822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1572.109045] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1572.189896] Lustre: lustre-MDT0000: new disk, initializing [ 1572.270296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1577.472596] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1589.129610] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1596.459231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1596.822279] Lustre: lustre-OST0000: new disk, initializing [ 1596.827350] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1596.831231] Lustre: Skipped 1 previous similar message [ 1596.836784] Lustre: 54926:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1598.048056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1598.062781] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1598.104083] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1603.645751] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1613.705888] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1616.225839] Lustre: DEBUG MARKER: create test files [ 1657.234766] Lustre: server umount lustre-OST0000 complete [ 1670.458337] Lustre: server umount lustre-MDT0000 complete [ 1685.817515] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 1689.167295] Key type lgssc unregistered [ 1689.681939] LNet: 56565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1689.691151] LNetError: 56565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1689.722701] LNet: Removed LNI 192.168.204.101@tcp [ 1691.069275] Key type .llcrypt unregistered [ 1691.074033] Key type ._llcrypt unregistered [ 1723.424079] Key type ._llcrypt registered [ 1723.425642] Key type .llcrypt registered [ 1723.587265] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 1724.758989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1724.860726] alg: No test for adler32 (adler32-zlib) [ 1725.951704] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1726.237257] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 1727.920315] Key type lgssc registered [ 1729.056771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1738.685301] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1738.721632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1740.222121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1745.101455] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1754.979097] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1762.500574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1762.947161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1766.000973] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:101 to 0x240000400:129) [ 1766.086436] Lustre: 59299:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 511 < left 544, rollback = 9 [ 1766.096310] Lustre: 59299:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/4, destroy: 0/0/0 [ 1766.100966] Lustre: 59299:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1766.106405] Lustre: 59299:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1766.119250] Lustre: 59299:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1766.124434] Lustre: 59299:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1770.149959] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1779.240421] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1785.835340] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1792.214503] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1801.685597] Lustre: server umount lustre-OST0000 complete [ 1815.755898] Lustre: server umount lustre-MDT0000 complete [ 1828.276249] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 1830.910790] Key type lgssc unregistered [ 1831.249776] LNet: 60910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1831.257627] LNetError: 60910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1831.275828] LNet: Removed LNI 192.168.204.101@tcp [ 1832.169041] Key type .llcrypt unregistered [ 1832.172652] Key type ._llcrypt unregistered [ 1863.513429] Key type ._llcrypt registered [ 1863.520458] Key type .llcrypt registered [ 1863.670947] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 1864.809731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1864.953435] alg: No test for adler32 (adler32-zlib) [ 1866.051261] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1866.366658] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 1868.064160] Key type lgssc registered [ 1869.396739] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1878.946360] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1878.995928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1880.417265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1885.335912] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1892.664545] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1899.138377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1899.452088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1905.581285] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1905.708340] Lustre: 63745:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1905.713623] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 1905.721615] Lustre: 63745:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1905.721631] Lustre: 63745:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1905.721637] Lustre: 63745:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1905.721644] Lustre: 63745:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1905.721649] Lustre: 63745:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1913.431304] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1919.166802] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1931.235493] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1931.250476] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1931.269282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1935.456448] Lustre: server umount lustre-OST0000 complete [ 1941.136732] Lustre: server umount lustre-MDT0000 complete [ 1956.774077] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 1960.387734] Key type lgssc unregistered [ 1960.779161] LNet: 65310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1960.786062] LNetError: 65310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1960.804642] LNet: Removed LNI 192.168.204.101@tcp [ 1961.703456] Key type .llcrypt unregistered [ 1961.705838] Key type ._llcrypt unregistered [ 1980.564050] Key type ._llcrypt registered [ 1980.566408] Key type .llcrypt registered [ 1980.703139] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [ 1997.419546] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 1998.227431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1998.341722] alg: No test for adler32 (adler32-zlib) [ 1999.723189] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1999.971245] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 2001.616284] Key type lgssc registered [ 2002.753714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2041.303383] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2041.330734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2042.635684] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2042.685740] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2042.825387] Lustre: lustre-MDT0000: new disk, initializing [ 2042.913924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2042.931748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2048.028923] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2059.944442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2060.071327] Lustre: 68464:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2060.121581] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2060.125049] Lustre: Skipped 1 previous similar message [ 2060.186186] Lustre: lustre-MDT0001: new disk, initializing [ 2060.275579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2060.331899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2060.337865] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2065.413099] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2076.513540] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2083.712713] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2091.864713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2092.354280] Lustre: lustre-OST0000: new disk, initializing [ 2092.362429] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2092.379526] Lustre: 70432:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2092.507381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2098.235376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2098.246401] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2098.371924] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2099.665291] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2111.407208] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2117.568296] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2117.839238] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2122.047597] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2122.197681] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2129.378142] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2129.380589] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2129.409619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2130.667535] Lustre: server umount lustre-OST0000 complete [ 2137.069258] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2137.081828] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2137.097294] Lustre: Skipped 1 previous similar message [ 2137.120364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2137.145655] Lustre: Skipped 1 previous similar message [ 2139.618791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2139.622745] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2142.113953] Lustre: server umount lustre-MDT0000 complete [ 2146.057625] LustreError: 68455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443900 with bad export cookie 6312985170404571949 [ 2146.078641] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2146.538306] Lustre: server umount lustre-MDT0001 complete [ 2159.634760] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 09:58:31 (1788443911) [ 2171.298273] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 2218.349203] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [ 2230.064322] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 2271.331038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2271.724917] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2271.766631] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2271.896980] Lustre: lustre-MDT0000: new disk, initializing [ 2272.022657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2272.055067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2276.988405] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2288.660640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2288.818678] Lustre: 76457:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2288.871540] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2288.875291] Lustre: Skipped 1 previous similar message [ 2288.979726] Lustre: lustre-MDT0001: new disk, initializing [ 2289.071558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2289.110865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2289.127803] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2293.938740] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2304.007411] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2311.144502] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2320.378288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2320.682586] Lustre: lustre-OST0000: new disk, initializing [ 2320.686803] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2320.696283] Lustre: 78439:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2320.778344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2328.189217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2328.227659] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2328.418666] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2331.210255] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2341.472590] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2346.740230] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2347.030654] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2352.252661] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2352.391502] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2359.272530] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2359.286502] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2359.295971] LustreError: Skipped 1 previous similar message [ 2359.300820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2359.305210] Lustre: Skipped 1 previous similar message [ 2359.316934] Lustre: Skipped 2 previous similar messages [ 2360.993466] Lustre: server umount lustre-OST0000 complete [ 2365.946079] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2365.958337] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2365.995451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2371.589631] Lustre: server umount lustre-MDT0000 complete [ 2374.637133] LustreError: 78050:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2374.663234] LustreError: 78050:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2375.688831] LustreError: 76450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444129 with bad export cookie 6312985170404572831 [ 2375.694926] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2375.703201] LustreError: 76450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2376.208738] Lustre: server umount lustre-MDT0001 complete [ 2385.130985] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 10:02:17 (1788444137) [ 2392.108138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2392.497173] LustreError: 80853:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2392.560792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2397.475630] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2397.666814] LustreError: 80854:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2402.792157] LustreError: 80853:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2407.098826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2412.139644] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2419.605649] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2425.610163] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2434.327301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2434.643416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2434.648881] Lustre: Skipped 1 previous similar message [ 2443.225382] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2452.696429] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2456.651557] Lustre: 83524:0:(genops.c:1795:obd_export_evict_by_uuid()) MGS: evicting f9402187-a64e-485e-842a-aa0da93e14c1 at adminstrative request [ 2457.580462] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2457.598308] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2467.810791] Lustre: MGS: Client f9402187-a64e-485e-842a-aa0da93e14c1 (at 0@lo) reconnecting [ 2467.828956] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x579c3a909a46020a to 0x579c3a909a4604b1 [ 2467.853492] Lustre: MGC192.168.204.101@tcp: Connection restored to 0@lo (at 0@lo) [ 2495.356950] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2495.718513] Lustre: lustre-OST0001: new disk, initializing [ 2495.725029] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2495.733697] Lustre: 83920:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2502.259972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2502.279128] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2502.435052] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2503.891304] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2515.478472] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2521.086770] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2521.107767] Lustre: Skipped 1 previous similar message [ 2521.117679] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2521.126656] Lustre: Skipped 3 previous similar messages [ 2525.593160] Lustre: server umount lustre-OST0001 complete [ 2526.196237] LustreError: 82676:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2526.223815] LustreError: 82676:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2530.277122] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2536.423325] LustreError: 82675:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2536.439033] LustreError: 82675:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2536.545218] Lustre: server umount lustre-OST0000 complete [ 2541.537475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2541.554065] Lustre: Skipped 3 previous similar messages [ 2541.557476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2541.560583] Lustre: Skipped 4 previous similar messages [ 2555.360199] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2555.697752] Lustre: server umount lustre-MDT0000 complete [ 2556.905387] LustreError: 83582:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2556.929115] LustreError: 83582:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2560.278036] LustreError: 83919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444314 with bad export cookie 6312985170404574385 [ 2560.284663] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2560.292726] LustreError: 83919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2560.849895] Lustre: server umount lustre-MDT0001 complete [ 2572.969418] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 10:05:24 (1788444324) [ 2575.417802] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2577.478046] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 10:05:29 (1788444329) [ 2579.676988] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2581.533931] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 10:05:33 (1788444333) [ 2583.328370] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2586.053397] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 10:05:37 (1788444337) [ 2588.388324] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2590.679136] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 10:05:42 (1788444342) [ 2593.059606] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2600.621806] loop: module loaded [ 2600.654322] loop0: detected capacity change from 0 to 4294967296 [ 2600.693578] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2602.688734] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2606.805125] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2611.396175] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2644.132903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2644.174634] LustreError: 88049:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2644.184628] LustreError: 88049:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2644.207655] LustreError: 88049:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2644.217212] LustreError: 88049:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2667.358518] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [ 2679.938973] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 2721.645664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2721.943105] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2721.978356] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2722.080617] Lustre: lustre-MDT0000: new disk, initializing [ 2722.131505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2722.135677] Lustre: Skipped 1 previous similar message [ 2722.145715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2726.266464] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2739.902762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2740.049177] Lustre: 90913:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2740.324711] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2746.788212] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2757.514667] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2764.773497] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2773.566669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2773.940050] Lustre: 92884:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2781.520217] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2783.271208] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2783.383762] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2793.116048] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2799.048900] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2799.352105] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2805.052876] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2805.342645] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2808.810968] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2808.832302] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2808.861994] Lustre: Skipped 1 previous similar message [ 2808.874791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2808.880976] Lustre: Skipped 4 previous similar messages [ 2813.918474] Lustre: server umount lustre-OST0000 complete [ 2824.584785] Lustre: server umount lustre-MDT0000 complete [ 2824.682249] LustreError: 92478:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2824.713239] LustreError: 92478:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2829.367631] LustreError: 91711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444583 with bad export cookie 6312985170404575001 [ 2829.370088] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2829.375252] LustreError: 91711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2829.849261] Lustre: server umount lustre-MDT0001 complete [ 2840.632631] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 10:09:52 (1788444592) [ 2848.882037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2855.067161] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2864.679627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2865.075671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-60 [ 2865.089744] Lustre: Skipped 3 previous similar messages [ 2870.352506] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2879.295601] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2885.944805] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2894.228806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2901.044122] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2908.619129] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2914.141644] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2914.539524] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2920.373720] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2920.773299] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2930.085206] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2930.242299] Lustre: lustre-OST0001: new disk, initializing [ 2930.245414] Lustre: Skipped 2 previous similar messages [ 2930.248886] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2930.253835] Lustre: Skipped 3 previous similar messages [ 2930.258710] Lustre: 98666:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2937.976514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2937.986948] Lustre: Skipped 2 previous similar messages [ 2937.996586] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2938.094672] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2938.129289] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 2948.338077] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2954.497884] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2954.712389] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2959.801971] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2960.153516] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2981.131490] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3008.156247] Lustre: Failing over lustre-MDT0000 [ 3008.480974] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3008.485730] LustreError: Skipped 1 previous similar message [ 3008.489992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3008.500881] Lustre: Skipped 3 previous similar messages [ 3008.505475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3008.510330] Lustre: Skipped 4 previous similar messages [ 3008.711149] Lustre: server umount lustre-MDT0000 complete [ 3009.531957] LustreError: 97133:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3009.557362] LustreError: 97133:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3026.401087] Lustre: 66689:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788444763/real 1788444763] req@0000000070adfc17 x1875319197035264/t0(0) o400->MGC192.168.204.101@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788444779 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3026.418386] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3037.732376] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 3037.736278] LDISKFS-fs (dm-0): recovery complete [ 3037.744934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3052.003872] LustreError: 66685:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000107957da x1875319197048576/t0(0) o250->MGC192.168.204.101@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3052.577542] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3055.558832] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3055.560588] Lustre: 95291:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 3057.645272] Lustre: 97470:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 3057.654794] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3057.663320] Lustre: 97470:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 1 previous similar message [ 3057.767793] LustreError: 101526:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3058.126751] LustreError: 101526:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3058.327962] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3058.462086] LustreError: 101526:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3058.489487] LustreError: 101526:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 1 previous similar message [ 3058.816185] LustreError: 101526:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3058.822354] LustreError: 101526:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 1 previous similar message [ 3058.837642] Lustre: 101526:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3058.852885] Lustre: 101526:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 7 previous similar messages [ 3059.505887] LustreError: 101526:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3059.518841] LustreError: 101526:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 2 previous similar messages [ 3059.848118] LustreError: 101526:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3059.859825] LustreError: 101526:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 2 previous similar messages [ 3061.156909] Lustre: 101526:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3061.175715] Lustre: 101526:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 6 previous similar messages [ 3061.515832] LustreError: 101526:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3061.541584] LustreError: 101526:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 5 previous similar messages [ 3061.888661] LustreError: 101526:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3061.900494] LustreError: 101526:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 5 previous similar messages [ 3065.433766] Lustre: 101526:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3065.457993] Lustre: 101526:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 12 previous similar messages [ 3065.793935] LustreError: 101526:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3065.811234] LustreError: 101526:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 12 previous similar messages [ 3066.149371] LustreError: 101526:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3066.159135] LustreError: 101526:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 12 previous similar messages [ 3073.514635] Lustre: 101526:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3073.538556] Lustre: 101526:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 24 previous similar messages [ 3073.865939] LustreError: 101526:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3073.890040] LustreError: 101526:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 24 previous similar messages [ 3074.216186] LustreError: 101526:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3074.235169] LustreError: 101526:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 24 previous similar messages [ 3089.549055] Lustre: 101526:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3089.564952] Lustre: 101526:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 49 previous similar messages [ 3089.897721] LustreError: 101526:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3089.931613] LustreError: 101526:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 49 previous similar messages [ 3090.257491] LustreError: 101526:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3090.283704] LustreError: 101526:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 49 previous similar messages [ 3115.298709] Lustre: 101526:0:(ldlm_lib.c:2394:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 3115.303628] Lustre: 101526:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3115.307608] Lustre: 101526:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 4b590548-7098-4067-a5cf-03c39e38fee7@192.168.204.1@tcp [ 3115.314055] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 3115.317345] Lustre: 101526:0:(ldlm_lib.c:1914:abort_req_replay_queue()) @@@ aborted: req@00000000c3ad5aef x1875319180038912/t0(8589934775) o101->4b590548-7098-4067-a5cf-03c39e38fee7@192.168.204.1@tcp:124/0 lens 592/0 e 0 to 0 dl 1788444879 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 3115.334637] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3115.347414] Lustre: lustre-MDT0000: Denying connection for new client 4b590548-7098-4067-a5cf-03c39e38fee7 (at 192.168.204.1@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 3115.348264] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3115.381751] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3115.385692] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 3115.386652] Lustre: Skipped 2 previous similar messages [ 3115.411962] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:47 to 0x2c0000401:545) [ 3115.412649] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:78 to 0x280000401:577) [ 3139.560946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3139.568653] Lustre: Skipped 2 previous similar messages [ 3147.232136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3147.377589] Lustre: server umount lustre-OST0000 complete [ 3149.805950] LustreError: 97185:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3149.846484] LustreError: 97185:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 59 previous similar messages [ 3153.378084] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3153.384256] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3153.397476] Lustre: Skipped 5 previous similar messages [ 3167.200688] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3167.423809] Lustre: server umount lustre-OST0001 complete [ 3178.973897] Lustre: server umount lustre-MDT0000 complete [ 3184.297177] LustreError: 102406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444938 with bad export cookie 6312985170404626220 [ 3184.312114] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3184.319199] LustreError: 102406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3185.134592] Lustre: server umount lustre-MDT0001 complete [ 3195.986117] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 10:15:47 (1788444947) [ 3204.919933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3204.938331] Lustre: *** cfs_fail_loc=197, val=0*** [ 3220.833074] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3225.956197] LustreError: 103504:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.101@tcp: failed processing log, type 4: rc = -110 [ 3231.111888] Lustre: *** cfs_fail_loc=197, val=0*** [ 3231.128963] Lustre: Skipped 65 previous similar messages [ 3251.680452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3251.686297] Lustre: Skipped 3 previous similar messages [ 3258.069665] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3268.247057] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3271.288800] Lustre: server umount lustre-OST0000 complete [ 3283.041324] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 10:17:14 (1788445034) [ 3306.165399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3312.540960] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3323.787902] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3328.134617] LustreError: 105155:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3328.149422] LustreError: 105155:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3328.163490] LustreError: 105155:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 3328.772832] Lustre: server umount lustre-MDT0000 complete [ 3366.533608] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [ 3378.914222] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 3421.793963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3421.980339] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3421.997606] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3422.053757] Lustre: lustre-MDT0000: new disk, initializing [ 3422.113282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3426.975162] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3437.085563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3437.178727] Lustre: 109332:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3437.452535] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3443.137411] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3453.038383] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3460.830562] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3468.930162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3469.394670] Lustre: 111304:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3476.790619] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3477.972886] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3488.827225] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3494.328349] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3494.646826] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3500.629886] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3500.826719] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3503.589075] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3503.594431] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3503.596486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3503.596495] Lustre: Skipped 12 previous similar messages [ 3503.605064] LustreError: Skipped 1 previous similar message [ 3503.650885] Lustre: Skipped 4 previous similar messages [ 3509.312845] Lustre: server umount lustre-OST0000 complete [ 3520.198326] Lustre: server umount lustre-MDT0000 complete [ 3524.066727] LustreError: 109346:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3524.087436] LustreError: 109346:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 3524.326723] LustreError: 111314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445278 with bad export cookie 6312985170404636853 [ 3524.327524] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3524.337689] LustreError: 111314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3524.849940] Lustre: server umount lustre-MDT0001 complete [ 3534.676676] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 10:21:26 (1788445286) [ 3543.769159] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 3547.063799] Key type lgssc unregistered [ 3547.410374] LNet: 113793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3547.421944] LNetError: 113793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3547.435649] LNet: Removed LNI 192.168.204.101@tcp [ 3548.445218] Key type .llcrypt unregistered [ 3548.446962] Key type ._llcrypt unregistered [ 3569.381671] Key type ._llcrypt registered [ 3569.386268] Key type .llcrypt registered [ 3569.485405] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [ 3590.085455] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 3591.875520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3591.956482] alg: No test for adler32 (adler32-zlib) [ 3593.063686] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3593.270191] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 3594.957541] Key type lgssc registered [ 3596.332951] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3647.761944] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3647.814185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3649.217352] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3649.309861] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3649.437602] Lustre: lustre-MDT0000: new disk, initializing [ 3649.578804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3649.602955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3655.353423] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3667.544361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3667.946499] Lustre: lustre-OST0000: new disk, initializing [ 3667.954509] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3667.961561] Lustre: Skipped 1 previous similar message [ 3667.970373] Lustre: 117969:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3668.071794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3674.227600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3674.237119] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3674.259197] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3674.644857] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3686.353037] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3686.670896] Lustre: lustre-OST0001: new disk, initializing [ 3686.676719] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3686.693916] Lustre: 118833:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3686.875417] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3693.503798] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3694.236704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3694.265266] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3694.322063] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3708.075080] Lustre: server umount lustre-MDT0000 complete [ 3717.213746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3722.102849] LustreError: 117155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445476 with bad export cookie 6626508915131430598 [ 3722.112494] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3725.792175] Lustre: 115173:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445463/real 1788445463] req@0000000013b2d85c x1875320868202368/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788445479 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3725.809311] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3730.848172] Lustre: 115171:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445468/real 1788445468] req@0000000050da2b11 x1875320868202752/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788445484 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3733.120863] Lustre: server umount lustre-OST0000 complete [ 3736.032786] Lustre: 115171:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445473/real 1788445473] req@00000000530a095d x1875320868203136/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788445489 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3737.469640] Lustre: server umount lustre-OST0001 complete [ 3747.970653] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 10:24:59 (1788445499) [ 3773.004681] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 10:25:25 (1788445525) [ 3784.811635] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 3811.476524] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 10:26:03 (1788445563) [ 3852.916535] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [ 3862.602303] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 3910.168268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3910.537818] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3910.602400] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3910.782183] Lustre: lustre-MDT0000: new disk, initializing [ 3910.881887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3910.898687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3916.012874] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3928.945770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3929.090847] Lustre: 126592:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3929.138743] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3929.142863] Lustre: Skipped 1 previous similar message [ 3929.206419] Lustre: lustre-MDT0001: new disk, initializing [ 3929.303829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3929.379718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3929.397877] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3935.870633] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3947.568370] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3955.019700] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3963.185992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3963.448954] Lustre: lustre-OST0000: new disk, initializing [ 3963.455908] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3963.467134] Lustre: 128562:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3963.580579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3969.670993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3969.694409] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3969.842648] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3970.921161] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 3981.780713] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3988.208481] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3988.493090] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3994.369762] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3994.675400] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4000.739161] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4000.749168] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4000.751431] LustreError: Skipped 1 previous similar message [ 4000.786159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4003.458454] Lustre: server umount lustre-OST0000 complete [ 4010.978437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4010.999919] Lustre: Skipped 1 previous similar message [ 4011.007061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4011.015121] Lustre: Skipped 2 previous similar messages [ 4014.655319] Lustre: server umount lustre-MDT0000 complete [ 4019.084164] LustreError: 128561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445773 with bad export cookie 6626508915131431823 [ 4019.086331] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4019.096102] LustreError: 128561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4019.494800] Lustre: server umount lustre-MDT0001 complete [ 4029.567606] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 10:29:41 (1788445781) [ 4037.544564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4037.933948] LustreError: 130973:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4037.951026] LustreError: 130973:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4037.999402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4042.758625] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4043.241583] LustreError: 130974:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4047.341955] LustreError: 130973:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4052.406494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4052.451600] LustreError: 130974:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4057.860499] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4067.113918] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4074.523568] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4082.743314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4083.315636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4083.330186] Lustre: Skipped 1 previous similar message [ 4091.424794] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4101.206965] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4117.679212] Lustre: *** cfs_fail_loc=159, val=0*** [ 4133.358239] Lustre: lustre-MDT0000: Client 511dc82c-ef7e-495c-a562-1932a83523ff (at 192.168.204.1@tcp) reconnecting [ 4135.134416] Lustre: *** cfs_fail_loc=159, val=0*** [ 4135.145561] Lustre: Skipped 7 previous similar messages [ 4150.779353] Lustre: lustre-MDT0000: Client 511dc82c-ef7e-495c-a562-1932a83523ff (at 192.168.204.1@tcp) reconnecting [ 4157.840864] Lustre: server umount lustre-OST0000 complete [ 4164.586102] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4164.593776] Lustre: Skipped 2 previous similar messages [ 4164.596845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4164.606527] Lustre: Skipped 1 previous similar message [ 4169.713785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4169.723240] Lustre: Skipped 1 previous similar message [ 4172.910022] Lustre: server umount lustre-MDT0000 complete [ 4174.817438] LustreError: 130992:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4174.848715] LustreError: 130992:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4175.850160] Lustre: 115172:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445913/real 1788445913] req@0000000095e97e93 x1875320868294144/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788445929 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4175.852873] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4175.881319] Lustre: 115172:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4177.554775] LustreError: 130955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445931 with bad export cookie 6626508915131432705 [ 4177.558572] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4177.567290] LustreError: 130955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4178.190222] Lustre: server umount lustre-MDT0001 complete [ 4193.424408] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 4197.987546] Key type lgssc unregistered [ 4198.438725] LNet: 134781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4198.477088] LNetError: 134781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4198.509838] LNet: Removed LNI 192.168.204.101@tcp [ 4200.054252] Key type .llcrypt unregistered [ 4200.059879] Key type ._llcrypt unregistered [ 4225.456160] Key type ._llcrypt registered [ 4225.460263] Key type .llcrypt registered [ 4225.630867] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 4227.096982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4227.123759] alg: No test for adler32 (adler32-zlib) [ 4228.266977] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 4228.607213] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 4230.312191] Key type lgssc registered [ 4231.417136] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4239.392723] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4239.410527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4240.908430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4245.582342] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4250.090958] LustreError: 136197:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4250.117962] LustreError: 136197:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4254.496750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4254.894599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4259.749082] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4266.936882] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4273.356849] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4280.327955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4280.699848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4285.818558] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4287.310278] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4297.363596] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4309.118151] Lustre: *** cfs_fail_loc=159, val=0*** [ 4324.345276] Lustre: lustre-MDT0000: Client 1f8566b3-26c4-4265-b7c0-9f171a4f05dd (at 192.168.204.1@tcp) reconnecting [ 4326.731633] Lustre: *** cfs_fail_loc=159, val=0*** [ 4342.280779] Lustre: lustre-MDT0001: Client 1f8566b3-26c4-4265-b7c0-9f171a4f05dd (at 192.168.204.1@tcp) reconnecting [ 4344.098729] Lustre: *** cfs_fail_loc=159, val=0*** [ 4344.104479] Lustre: Skipped 3 previous similar messages [ 4359.707288] Lustre: lustre-MDT0001: Client 1f8566b3-26c4-4265-b7c0-9f171a4f05dd (at 192.168.204.1@tcp) reconnecting [ 4365.493340] Lustre: 139233:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4368.990836] Lustre: *** cfs_fail_loc=159, val=0*** [ 4368.993117] Lustre: Skipped 4 previous similar messages [ 4384.281348] Lustre: lustre-MDT0000: Client 6d73b3c3-253a-4a3a-88c3-444a31a9f4b8 (at 192.168.204.1@tcp) reconnecting [ 4386.164302] Lustre: *** cfs_fail_loc=159, val=0*** [ 4386.166199] Lustre: Skipped 13 previous similar messages [ 4401.704545] Lustre: lustre-MDT0000: Client 6d73b3c3-253a-4a3a-88c3-444a31a9f4b8 (at 192.168.204.1@tcp) reconnecting [ 4403.177969] Lustre: 139527:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4422.113621] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4422.128130] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4422.135152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4422.143085] Lustre: Skipped 1 previous similar message [ 4422.157721] Lustre: Skipped 1 previous similar message [ 4427.233515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4427.251830] Lustre: Skipped 1 previous similar message [ 4428.102726] Lustre: server umount lustre-OST0000 complete [ 4433.891782] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4433.896943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4433.915357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4437.473601] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4437.474815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4437.502729] Lustre: Skipped 1 previous similar message [ 4442.594995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4442.602678] Lustre: Skipped 1 previous similar message [ 4447.201048] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4447.356634] Lustre: server umount lustre-MDT0000 complete [ 4447.714377] LustreError: 136189:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4447.734552] LustreError: 136189:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4451.226144] LustreError: 139357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446205 with bad export cookie 3450105392726232933 [ 4451.236063] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4451.245317] LustreError: 139357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4451.722189] Lustre: server umount lustre-MDT0001 complete [ 4462.405597] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 4465.466960] Key type lgssc unregistered [ 4465.798386] LNet: 140610:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4465.807873] LNetError: 140610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4465.824828] LNet: Removed LNI 192.168.204.101@tcp [ 4466.838810] Key type .llcrypt unregistered [ 4466.840554] Key type ._llcrypt unregistered [ 4494.495389] Key type ._llcrypt registered [ 4494.501344] Key type .llcrypt registered [ 4494.639427] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 4495.870924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4495.928787] alg: No test for adler32 (adler32-zlib) [ 4497.104485] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 4497.372470] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 4499.017477] Key type lgssc registered [ 4500.229094] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4509.284918] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4509.301900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4510.932183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4515.794263] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4520.427199] LustreError: 142026:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4520.452110] LustreError: 142026:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4525.552658] LustreError: 142025:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4525.695739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4526.099629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4530.604951] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4538.615343] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4544.752443] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4551.855882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4552.251427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4555.320755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4558.819443] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4566.579143] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4570.734066] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4575.916844] Lustre: 144610:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4578.844847] Lustre: 144658:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4580.226629] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4581.059784] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4584.981946] Lustre: 144804:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4585.857490] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4586.784661] Lustre: 144903:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4591.075050] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4591.083059] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4591.106103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4591.587800] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4591.635471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4596.721527] LustreError: 143847:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4596.746844] LustreError: 143847:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4596.846494] Lustre: server umount lustre-OST0000 complete [ 4601.829190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4601.857689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4601.867512] Lustre: Skipped 1 previous similar message [ 4606.951166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4606.958663] Lustre: Skipped 1 previous similar message [ 4607.587413] Lustre: server umount lustre-MDT0000 complete [ 4611.604091] LustreError: 142007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446365 with bad export cookie 9854717693899410992 [ 4611.615075] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4611.618994] LustreError: 142007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4612.040850] Lustre: server umount lustre-MDT0001 complete [ 4622.998782] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 4626.150551] Key type lgssc unregistered [ 4626.527853] LNet: 145966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4626.542434] LNetError: 145966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4626.582848] LNet: Removed LNI 192.168.204.101@tcp [ 4627.707832] Key type .llcrypt unregistered [ 4627.711340] Key type ._llcrypt unregistered [ 4655.383199] Key type ._llcrypt registered [ 4655.385979] Key type .llcrypt registered [ 4655.497916] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 4656.678729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4656.695684] alg: No test for adler32 (adler32-zlib) [ 4657.863372] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 4658.160979] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 4659.864183] Key type lgssc registered [ 4661.052512] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4669.022806] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4669.056488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4670.540396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4675.032824] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4680.682448] LustreError: 147380:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4680.705296] LustreError: 147380:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4682.821398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4683.077532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4687.321890] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4693.975072] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4699.611505] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4706.422852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4706.823554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4707.919630] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4713.387055] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4716.084520] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4722.293800] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4728.502106] Lustre: *** cfs_fail_loc=159, val=0*** [ 4744.710609] Lustre: lustre-MDT0000: Client c8b4b0e2-7226-4303-9dbe-823ed7dc3d86 (at 192.168.204.1@tcp) reconnecting [ 4750.038958] Lustre: server umount lustre-OST0000 complete [ 4754.916645] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4754.930182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4754.952381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4756.961475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4756.980762] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4762.082348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4762.084369] Lustre: Skipped 1 previous similar message [ 4763.192266] Lustre: server umount lustre-MDT0000 complete [ 4767.100729] LustreError: 147361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446521 with bad export cookie 17339229663401107255 [ 4767.110553] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4777.701928] Lustre: server umount lustre-MDT0001 complete [ 4789.924633] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 4793.282769] Key type lgssc unregistered [ 4793.714858] LNet: 151056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4793.730921] LNetError: 151056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4793.761559] LNet: Removed LNI 192.168.204.101@tcp [ 4794.809386] Key type .llcrypt unregistered [ 4794.812938] Key type ._llcrypt unregistered [ 4821.359655] Key type ._llcrypt registered [ 4821.361949] Key type .llcrypt registered [ 4821.483716] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 4822.937909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4822.970835] alg: No test for adler32 (adler32-zlib) [ 4824.109738] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 4824.412581] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 4826.048192] Key type lgssc registered [ 4827.224961] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4836.269345] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4836.299029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4837.769928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4841.933688] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4847.080316] LustreError: 152478:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4847.101383] LustreError: 152478:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4850.696326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4851.042283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4855.114768] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4861.173428] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4865.806287] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4872.591047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4872.854547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4875.957043] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4878.297595] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 4882.099467] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4884.920737] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4904.387770] Lustre: lustre-OST0000: evicting 77374de8-ff56-488a-8314-8deb173870a6 (at 192.168.204.1@tcp) by administrative request [ 4904.396941] Lustre: Skipped 1 previous similar message [ 4933.601855] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4933.620587] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4933.637192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4936.691520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4936.697106] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4939.705771] Lustre: server umount lustre-OST0000 complete [ 4946.917182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4946.922420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4946.928952] Lustre: Skipped 1 previous similar message [ 4946.937955] Lustre: Skipped 1 previous similar message [ 4950.308865] Lustre: server umount lustre-MDT0000 complete [ 4952.043060] LustreError: 152475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4952.061594] LustreError: 152475:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4954.039397] LustreError: 152457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446707 with bad export cookie 13484039927115043739 [ 4954.045696] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4954.054759] LustreError: 152457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4954.389724] Lustre: server umount lustre-MDT0001 complete [ 4967.331137] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 4970.014650] Key type lgssc unregistered [ 4970.308668] LNet: 156552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4970.319393] LNetError: 156552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4970.340466] LNet: Removed LNI 192.168.204.101@tcp [ 4971.228252] Key type .llcrypt unregistered [ 4971.231312] Key type ._llcrypt unregistered [ 5039.326984] Key type ._llcrypt registered [ 5039.329792] Key type .llcrypt registered [ 5039.413521] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 5040.567218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5040.597328] alg: No test for adler32 (adler32-zlib) [ 5041.669626] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5041.935657] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 5043.624275] Key type lgssc registered [ 5044.721395] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5052.617293] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5052.635502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5053.982573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5059.261092] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5063.153479] LustreError: 159039:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5063.171658] LustreError: 159039:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5067.618255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5068.007046] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5073.234940] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5081.300721] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5087.244477] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5094.243184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5094.570380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5098.692988] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 5100.859711] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5109.026355] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5113.016214] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 5123.044241] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5123.051445] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5123.061749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5123.559644] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5123.567411] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5123.592174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5127.470696] Lustre: server umount lustre-OST0000 complete [ 5133.799182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5133.818065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5133.827435] Lustre: Skipped 1 previous similar message [ 5137.501886] Lustre: server umount lustre-MDT0000 complete [ 5138.914498] LustreError: 159040:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5138.934709] LustreError: 159040:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5141.256636] LustreError: 159021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446895 with bad export cookie 3883543014823435162 [ 5141.258852] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5141.267733] LustreError: 159021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5141.565940] Lustre: server umount lustre-MDT0001 complete [ 5150.641491] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 5153.276302] Key type lgssc unregistered [ 5153.546076] LNet: 162597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5153.555132] LNetError: 162597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5153.564857] LNet: Removed LNI 192.168.204.101@tcp [ 5154.279501] Key type .llcrypt unregistered [ 5154.281533] Key type ._llcrypt unregistered [ 5175.551086] Key type ._llcrypt registered [ 5175.554739] Key type .llcrypt registered [ 5175.666969] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 5176.543698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5176.632066] alg: No test for adler32 (adler32-zlib) [ 5177.867541] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5178.142836] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 5179.832327] Key type lgssc registered [ 5180.618757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5195.146653] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 10:49:07 (1788446947) [ 5211.578695] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [ 5220.507246] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 5254.248960] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5254.289112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5255.541645] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5255.601184] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5255.699957] Lustre: lustre-MDT0000: new disk, initializing [ 5255.791205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5255.805896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5259.991196] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5270.436916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5270.570360] Lustre: 166854:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5270.626523] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5270.630976] Lustre: Skipped 1 previous similar message [ 5270.697710] Lustre: lustre-MDT0001: new disk, initializing [ 5270.792480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5270.855346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5270.866421] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5274.734645] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5283.721997] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5289.341687] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5296.401864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5296.748418] Lustre: lustre-OST0000: new disk, initializing [ 5296.760598] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5296.774864] Lustre: 168824:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5296.912506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5302.501943] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5303.899992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5303.909979] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5304.015143] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5311.255429] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5319.649426] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5319.651481] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5319.660300] LustreError: Skipped 1 previous similar message [ 5319.671129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5319.675857] Lustre: Skipped 1 previous similar message [ 5319.686415] Lustre: Skipped 1 previous similar message [ 5324.771254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5324.787300] Lustre: Skipped 1 previous similar message [ 5325.807505] Lustre: server umount lustre-OST0000 complete [ 5329.894206] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5329.897791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5329.909803] Lustre: Skipped 1 previous similar message [ 5335.009633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5335.015425] Lustre: Skipped 2 previous similar messages [ 5335.681579] Lustre: server umount lustre-MDT0000 complete [ 5339.633283] LustreError: 167650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447093 with bad export cookie 17063083810042894364 [ 5339.638912] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5339.657745] LustreError: 167650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5340.013546] Lustre: server umount lustre-MDT0001 complete [ 5350.302885] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 5354.377891] Key type lgssc unregistered [ 5354.801961] LNet: 170731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5354.816550] LNetError: 170731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5354.845217] LNet: Removed LNI 192.168.204.101@tcp [ 5355.605311] Key type .llcrypt unregistered [ 5355.607566] Key type ._llcrypt unregistered [ 5380.172433] Key type ._llcrypt registered [ 5380.174969] Key type .llcrypt registered [ 5380.255222] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 5381.243502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5381.413397] alg: No test for adler32 (adler32-zlib) [ 5382.458195] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5382.784694] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 5384.528655] Key type lgssc registered [ 5385.597516] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5394.349374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5394.382486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5395.904777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5399.945945] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5405.163915] LustreError: 172147:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5405.185762] LustreError: 172147:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5408.705089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5409.167172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5413.921795] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5421.869669] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5427.761687] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5434.824624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5435.218101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5440.380990] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5441.358512] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 5449.388597] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5459.059063] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5459.086885] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5459.112344] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5459.121661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5459.137973] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5460.153872] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5460.172965] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5460.183753] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5460.192214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5460.203288] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5461.209226] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5461.241141] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5461.267991] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5461.276769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.300657] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5462.395971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5463.277569] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5463.282350] Lustre: Skipped 1 previous similar message [ 5463.301781] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5463.311956] Lustre: Skipped 1 previous similar message [ 5463.316893] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5463.326739] Lustre: Skipped 1 previous similar message [ 5463.333115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5463.344631] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5463.352922] Lustre: Skipped 1 previous similar message [ 5464.432371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5465.446410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5466.663491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.741729] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5467.751626] Lustre: Skipped 3 previous similar messages [ 5467.786788] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5467.800982] Lustre: Skipped 3 previous similar messages [ 5467.811499] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5467.821703] Lustre: Skipped 3 previous similar messages [ 5467.829945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.841911] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5467.848350] Lustre: Skipped 3 previous similar messages [ 5468.953392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.782570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5470.751358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5471.661557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5472.660969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.719156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5474.789993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.673358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5476.578726] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5476.587434] Lustre: Skipped 8 previous similar messages [ 5476.601842] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5476.613355] Lustre: Skipped 8 previous similar messages [ 5476.619795] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5476.627328] Lustre: Skipped 8 previous similar messages [ 5476.631652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5476.641457] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5476.644175] Lustre: Skipped 8 previous similar messages [ 5477.562663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5478.560176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5479.719705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5480.740380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.710513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.490720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5483.336383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.153046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.022652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.822905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5486.708220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5487.688786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.691705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.576628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5490.437408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.282900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.278112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.106951] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5493.113630] Lustre: Skipped 17 previous similar messages [ 5493.135678] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5493.148871] Lustre: Skipped 17 previous similar messages [ 5493.163074] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5493.177915] Lustre: Skipped 17 previous similar messages [ 5493.185581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.201344] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5493.209570] Lustre: Skipped 17 previous similar messages [ 5494.003294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5494.784154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.592086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5496.342314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.217763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5498.066933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5498.884954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.652915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.570596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.386650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5502.265777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5503.155804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.015685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.925852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.842157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.786209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.747857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.678529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5509.581258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.502178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5511.494097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.418905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.476552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5514.714632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.898881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5516.721108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.606917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.565323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5519.564533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.436813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.391158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5522.302917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.268539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.207457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5525.200777] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5525.205922] Lustre: Skipped 34 previous similar messages [ 5525.219482] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5525.231260] Lustre: Skipped 34 previous similar messages [ 5525.234846] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5525.240316] Lustre: Skipped 34 previous similar messages [ 5525.243768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5525.250667] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5525.255608] Lustre: Skipped 34 previous similar messages [ 5526.311820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.302964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.204497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.114896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.956963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5530.881416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.820707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.539476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5533.339809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.166927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.942851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5535.742870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.588775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5537.544296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.457160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5539.265451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.175778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5541.031686] LustreError: 172167:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5541.041508] LustreError: 172167:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5541.063942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.037293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.038801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.821318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.714415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.538455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.409571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.306740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.148388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.058900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.000422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.873970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.862746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.649155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.505440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.626173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.479467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5556.246182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.008352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.849628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.614987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.382468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.103530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.800866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.535562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.441789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.186176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.898673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.692978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5565.453525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.266246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.241539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.098218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.880661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.696715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.683748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.430341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.279687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.170092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.133567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.891403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.954715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.750757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.536549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.396455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.322506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.284673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.292225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.055688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.864860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.679336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.492986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.376281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5586.444232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.246810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.970317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.780418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.664737] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5589.668623] Lustre: Skipped 74 previous similar messages [ 5589.687680] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5589.695592] Lustre: Skipped 74 previous similar messages [ 5589.699664] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5589.707848] Lustre: Skipped 74 previous similar messages [ 5589.716213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.729150] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5589.742173] Lustre: Skipped 74 previous similar messages [ 5590.490706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.307701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.134628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.964679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.862820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.709982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.712219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.694696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.589340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.459898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.373637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.333958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.251157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.029758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.016773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.995091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.855259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.618675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.313839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.042413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.759888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.641781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.468671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.179070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.083340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.863377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.736815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.677633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.514213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.349399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.282757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.182721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.046885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.941152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.873824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.645449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.444969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.282831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.145808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.996144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.815496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.561073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.371973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.241495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.067080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.740942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.530369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.324190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.141863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.073963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.855080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.738461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.616496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.476462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.553405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.515240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.453226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.356957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.527161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.518194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.543563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.436141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.360758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.180606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.085179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.873777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.640521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.409918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.265734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.977762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.804226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.551181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.356982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.157343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.147579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.001177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.870862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.687496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.505597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.342397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.158839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.114344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.916259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.623707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.488799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.213362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.138547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.014783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.060375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.228956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.131482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.955644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.757816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.784672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.865740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.746980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.885438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.767021] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.629757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.344508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.376118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.310120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.155975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.108931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.051876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.223902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.056346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.903195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.829891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.164085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.368908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.395368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.393267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.589698] LustreError: 172167:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5690.605287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.609144] LustreError: 172167:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5691.436174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.361651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.413248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.494742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.350471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.213642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.136835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.476652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.324454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.537691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.743704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.918251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.042050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.423282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.582475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.460692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.234187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.960657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.884331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.888235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.982941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5712.836456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.846314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.802149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.605079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.496669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.937467] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5717.950759] Lustre: Skipped 140 previous similar messages [ 5717.978343] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5717.992370] Lustre: Skipped 140 previous similar messages [ 5717.997728] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5718.002710] Lustre: Skipped 140 previous similar messages [ 5718.005826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.020563] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5718.026629] Lustre: Skipped 140 previous similar messages [ 5718.923585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.984779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.907465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.875700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.687265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.831104] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.915767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.806991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.760692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.801687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.916412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.822113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.986421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.167378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.401090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.330391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.426430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.378753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.564639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.724435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.764068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5740.826839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.062270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.163605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.248027] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.450297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.171236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.241778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.256257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.454985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5750.233833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.154505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.186967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.527962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.788534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.652704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.414405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.285175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.023180] LustreError: 172167:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5758.030551] LustreError: 172167:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5758.034516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.019786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.238993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.130853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.106724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.863399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.118672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.613713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.007591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.128813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.890771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.701898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.702459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.619132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.430902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.952925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.001665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.124804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.486652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.532124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.559785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.642171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.764931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.605568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.417191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.364637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.184795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.007251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.792206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.648529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.471503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.307081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.131643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.990647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.851585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.801820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.848981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.581938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.339196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.104555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.821616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.655314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.578361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.553080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.620294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.542465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.547627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.347982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.329847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.397852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.368593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.334481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.598306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.596677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.558806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.541538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.485937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.382634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5814.462884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.397884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.232402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.134652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.489671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.459478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.296850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.302759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.074547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.866929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.700462] LustreError: 172167:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5823.708983] LustreError: 172167:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5823.723953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.522904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.236206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.070107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.028471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.853279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.722858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.548772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.537650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.297046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.114477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.117162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.062652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5835.127458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.075636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.156507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.224733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.043779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.879281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.525623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.422465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.218525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.314294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5844.107542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.081439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.123270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.023280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.110609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.935161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.721162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.591525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.483730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.313039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.117654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.958817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.961444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.971336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.901219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5857.792867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.785859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.134886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.215524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.105968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.062662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.906823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.828125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.610177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.436750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.340152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.319802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.249229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.053262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.888058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.911174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.839984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.988530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.832592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5875.867729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.760765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.713727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.690785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.730529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.607766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5881.720561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5882.563982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.390201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.199563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.038545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.022168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.318868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.765410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.695518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.723391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.696623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.538847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.791942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.742814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.878245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.759679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.602075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.682452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.857631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.918980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.231616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.301937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.191718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.310743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.272426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.196108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.403784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.362821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.184356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.012639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.016494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.078207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.944609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.000433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.901558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.967618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.991631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5919.200271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.444699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5921.411705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.411494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5923.667846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.937028] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.152507] LustreError: 172167:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5926.167860] LustreError: 172167:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5926.180718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5927.746407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5928.961356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5930.124663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.077285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.231713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5933.062188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5933.924759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5934.907678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5935.763375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5936.755113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5937.633383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5938.582415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.494727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.460090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5941.273266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.141558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5943.083388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5944.161107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5945.178549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.356474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5947.432881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.447160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.419543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5950.179584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5950.926313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5951.752800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5952.485866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.418691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5954.399672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5955.244155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5956.086084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5957.077330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5958.097931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5958.934645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5959.684535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5960.547702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5961.283268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5962.138955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5963.031085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5964.048660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5965.085653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5966.223253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.433278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5968.532677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5969.481477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5970.271626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5971.027633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5971.773506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5972.716034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5973.944691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.017617] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5975.022312] Lustre: Skipped 263 previous similar messages [ 5975.043410] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5975.059471] Lustre: Skipped 263 previous similar messages [ 5975.067829] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5975.079405] Lustre: Skipped 263 previous similar messages [ 5975.086196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.098411] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5975.106868] Lustre: Skipped 263 previous similar messages [ 5975.941179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.888277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5977.726433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.706118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.507901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5980.548714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5981.500112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5982.608752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5983.520466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.640769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5985.437080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.478635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5987.915294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.791854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5989.830317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5990.822858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.751673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5992.721853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5993.667401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5994.621546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5995.577507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5996.349970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5997.214060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.341473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5999.048791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5999.882771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.677965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6001.781721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6002.664062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6003.826703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6004.969752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.066223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6007.118177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6008.026762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6008.959093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6010.072718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6010.923862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6011.793288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6012.675443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6013.577094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.577576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6015.431323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6016.260814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6017.131293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6017.972760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6018.996046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6019.783302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6020.682939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6021.398570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6022.383586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6023.530658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6024.591720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6025.376616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6026.409893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6027.435460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6028.533281] LustreError: 172167:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6028.549928] LustreError: 172167:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6028.563975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6029.404839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6030.433573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6031.153555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6032.121197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6033.047399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6033.852368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.751849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6035.762515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6036.709641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6037.675520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.647266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6039.845975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6041.105317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6042.532620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6043.744692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6044.927807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6046.429797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6047.794728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6049.101451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6050.257074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6051.492116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6052.535123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6053.381749] LustreError: 172167:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6053.392383] LustreError: 172167:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6053.402275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6054.426358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6055.537156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.584798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6057.450022] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6058.392551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6059.387966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6060.660300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6061.997492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6063.082135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.033211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6065.202798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6065.972315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6066.858273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6068.003530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6069.011903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6070.013180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6070.880116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6071.810485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6072.614866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6073.419814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6074.479484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.595736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6076.536583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6077.657673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6078.601078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6079.618385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6080.476901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6081.393233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6082.452546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6083.447687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6084.320954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6085.185619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6085.953896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6086.747053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6087.494139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6088.380074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6089.166540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6090.033746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6090.807323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6091.754602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6092.809245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6093.787740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6094.659867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6095.610290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6096.504323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6097.420822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6098.371568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6099.299050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6100.176768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6101.131400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6101.874872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6102.716111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6103.460138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6104.242464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6105.038518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6105.801656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6106.662839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6107.607658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6108.566912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6109.436339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6110.266631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6111.279899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6112.184542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6113.088706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6113.966637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6114.793638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6115.633319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6116.507845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6117.280629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6118.021343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6118.749087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6119.637701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6120.463286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6121.251518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6122.024325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6122.911963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6123.634476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6124.464313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6125.299159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6126.205253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6127.067936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6128.027902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6128.851333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6129.601830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6130.480349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6131.312049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6132.228405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6133.040462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6133.924507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6134.724184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6135.721853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6136.643633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6137.496745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6138.533746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6139.406923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6140.269473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6141.058956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6141.978334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6142.689859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6143.532510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6144.476629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6145.410174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6146.282417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6147.122785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6148.041486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6148.879751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6149.662903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6150.477050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6151.186766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6152.135865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6153.367590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6154.594241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6155.631428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6156.665778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6157.724780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6158.783707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6159.772066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6160.568984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6161.585703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6162.530963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6163.635361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6164.525202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6165.548802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6166.554900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6167.852297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6169.052529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6170.009604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6171.061319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6172.174769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6173.410559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6174.441126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6175.509883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6176.730712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6177.513859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6178.630451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6179.668174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6180.876701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6181.933277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6183.269438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6184.255467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6185.398100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6186.484823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6187.482840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6188.581786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6189.580849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6190.793287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6191.810221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6193.138081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6194.471785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6195.316103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6196.541243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6197.722733] LustreError: 172167:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6197.739653] LustreError: 172167:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6197.749513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6198.744559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6199.689157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6200.570224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6201.732278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6202.534505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6203.244489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6204.130191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6204.860302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6206.000637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6207.167083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6208.065170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6209.128294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6210.043471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6210.930905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6211.857892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6212.711067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6213.799769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6214.942388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6215.911279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6216.938457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6217.754542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6218.656827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6219.459059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6220.520902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6221.558888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6222.237623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6223.270472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6224.340395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6225.900425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6227.224604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6228.094589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6229.127087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6230.100311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6231.667080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6233.033895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6233.977515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6235.330885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6236.469308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6237.760523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6239.054536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6240.059711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6241.037471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6242.364932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6243.453088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6244.408968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6245.693361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6246.740054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6247.802171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6248.844559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6249.774199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6250.761549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6251.703528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6252.902672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6253.872549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6254.934608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6255.933720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6256.814535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6257.873492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6259.002985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6259.864244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6260.817380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6261.577058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6262.430410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6263.210113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6264.111569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6264.850730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6265.677788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6266.630987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6267.499411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6268.441559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6269.637490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6270.754542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6271.621531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6272.443804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6273.386057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6274.398271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6275.292740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6276.252201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6277.310636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6278.082124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6279.176459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6279.966814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6280.820926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6281.713262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6282.581124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6283.379164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6284.239745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6285.202581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6286.136530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6287.032493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6288.093024] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6289.000499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6289.908242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6290.824262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6291.833184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6292.662521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6293.524668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6294.373544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6295.402297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6296.181488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6296.983811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6297.902702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6298.760137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6299.470077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6300.206254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6300.958270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6301.756431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6302.467672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6303.399950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6304.307899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6305.108660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6305.886361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6306.711865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6307.601031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6308.558859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6309.450419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6310.218369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6311.045960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6311.994299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6312.774125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6313.591920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6314.449288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6315.221465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6316.079640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6316.832298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6317.610498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6318.412400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6319.193696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6319.987364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6320.818190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6321.938894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6322.680250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6323.741877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6324.798230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6325.717759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6326.685324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6327.823786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6328.820760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6329.781073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6330.765838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6331.770631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6332.641165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6333.447839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6334.239483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6334.985886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6335.574167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6336.353844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6337.176335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6338.040697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6338.948489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6339.778681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6340.671755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6341.615135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6342.399758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6343.242697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6344.167656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6345.138505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6345.928929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6346.903812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6347.675129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6348.627172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6350.058602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6351.212770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6352.240777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6353.448608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6354.699451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6355.964398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6356.946844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6358.438765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6359.635297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6360.802797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6361.657063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6362.456231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6363.344396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6364.216513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6365.224677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6366.308825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6367.264410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6368.489794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6369.596272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6370.714808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6371.667597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6372.666229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6373.700659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6374.791522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6375.784354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6376.622604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6377.723439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6378.655150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6379.562434] LustreError: 172167:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6379.574320] LustreError: 172167:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6379.584626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6380.529219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6381.675646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6382.672934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6384.068069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6385.287534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6386.256907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6387.378754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6388.481715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6389.487603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6390.550813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6391.591194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6392.443902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6393.343565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6394.429104] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6395.630634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6396.532973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6397.535881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6398.553631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6400.050652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6401.316794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6402.462464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6403.587960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6404.499410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6405.518719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6406.473902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6407.744972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6408.897495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6410.231235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6893.169971] LustreError: 173972:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6893.191530] LustreError: 172167:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 7669.733992] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7669.736841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7669.754255] Lustre: Skipped 450 previous similar messages [ 7669.765826] Lustre: Skipped 1 previous similar message [ 7673.516549] Lustre: server umount lustre-OST0000 complete [ 7679.971536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7679.985174] Lustre: Skipped 1 previous similar message [ 7684.026937] Lustre: server umount lustre-MDT0000 complete [ 7685.093839] LustreError: 226308:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7685.107474] LustreError: 226308:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7687.620122] LustreError: 172884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449441 with bad export cookie 8456103877342804146 [ 7687.622521] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7687.632299] LustreError: 172884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7687.988069] Lustre: server umount lustre-MDT0001 complete [ 7699.995636] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 7702.494178] Key type lgssc unregistered [ 7702.761742] LNet: 227449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7702.768524] LNetError: 227449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7702.792868] LNet: Removed LNI 192.168.204.101@tcp [ 7703.751079] Key type .llcrypt unregistered [ 7703.753450] Key type ._llcrypt unregistered [ 7730.507478] Key type ._llcrypt registered [ 7730.509237] Key type .llcrypt registered [ 7730.593630] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 7731.878241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7731.895789] alg: No test for adler32 (adler32-zlib) [ 7732.937381] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7733.163625] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 7734.865050] Key type lgssc registered [ 7736.181659] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7744.359453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7744.397470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7746.168801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7752.094616] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7756.776638] LustreError: 228864:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7756.792519] LustreError: 228864:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7761.455106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7761.987222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7767.884354] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7776.295696] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7781.946070] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7788.388533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7788.730849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7791.790766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 7795.260941] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7803.277799] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7812.401695] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7814.633376] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7814.643040] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 7814.658935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7814.677314] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7814.677954] LustreError: 228883:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 7821.284380] Lustre: lustre-OST0000: Not available for connect from 192.168.204.1@tcp (stopping) [ 7821.808677] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7821.820147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7821.843575] Lustre: Skipped 1 previous similar message [ 7821.854249] Lustre: Skipped 1 previous similar message [ 7823.301218] Lustre: server umount lustre-OST0000 complete [ 7834.704149] LustreError: 228883:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 7858.794915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7859.230529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7861.308330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7861.333140] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7861.345388] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7866.156803] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7874.309912] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7875.790292] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7882.084927] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff96fb476c5800.ost_server_uuid 50 [ 7884.131920] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff96fb476c5800.ost_server_uuid in IDLE state after 0 sec [ 7889.379628] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7889.387142] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7889.399751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7891.943056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7891.947411] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7894.525940] Lustre: server umount lustre-OST0000 complete [ 7900.130099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7900.145585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7900.159722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7900.164929] Lustre: Skipped 1 previous similar message [ 7905.715055] Lustre: server umount lustre-MDT0000 complete [ 7907.300193] LustreError: 228859:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7907.335263] LustreError: 228859:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7910.242630] LustreError: 228843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449664 with bad export cookie 203261166342639596 [ 7910.265771] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7910.279056] LustreError: 228843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7910.913352] Lustre: server umount lustre-MDT0001 complete [ 7930.567827] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 7933.773120] Key type lgssc unregistered [ 7934.120947] LNet: 234100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7934.131878] LNetError: 234100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7934.146702] LNet: Removed LNI 192.168.204.101@tcp [ 7935.464463] Key type .llcrypt unregistered [ 7935.467382] Key type ._llcrypt unregistered [ 7969.989319] Key type ._llcrypt registered [ 7969.991276] Key type .llcrypt registered [ 7970.071688] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 7971.527233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7971.588532] alg: No test for adler32 (adler32-zlib) [ 7972.756892] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7973.147344] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 7974.816333] Key type lgssc registered [ 7975.986765] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7985.429479] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7985.473656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7986.965606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7991.149202] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 7996.391309] LustreError: 235832:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7996.406424] LustreError: 235832:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7999.277996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7999.655364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8004.316735] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8007.167992] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8013.675135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8014.016328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8021.732836] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8031.419830] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 8031.594751] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8031.740756] Lustre: lustre-OST0001: new disk, initializing [ 8031.743819] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8031.749051] Lustre: 238139:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8031.831676] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8038.627599] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8040.113932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8040.136263] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8040.256470] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8051.419854] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8061.227634] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8076.772100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8076.794717] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8076.813363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8081.380570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8081.385435] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8081.401269] Lustre: Skipped 2 previous similar messages [ 8082.822878] Lustre: server umount lustre-MDT0000 complete [ 8086.370102] LustreError: 238966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449840 with bad export cookie 9674385676634968005 [ 8086.379745] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8086.385693] LustreError: 238966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8086.497981] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8086.506312] LustreError: 237792:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8086.516707] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8086.531888] LustreError: 237792:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8086.545474] Lustre: Skipped 4 previous similar messages [ 8086.687164] Lustre: server umount lustre-MDT0001 complete [ 8092.520519] Lustre: server umount lustre-OST0000 complete [ 8095.898166] Lustre: server umount lustre-OST0001 complete [ 8116.061736] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 8125.189192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8125.631923] LustreError: 241244:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8125.699050] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8130.996398] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8131.046046] LustreError: 241245:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8135.152736] LustreError: 241244:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8139.796556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8140.146061] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8144.488200] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8147.253317] Lustre: 242386:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8153.761787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8160.333625] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8165.478619] LustreError: 242742:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8168.399133] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8168.669491] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8168.682106] Lustre: Skipped 1 previous similar message [ 8170.772979] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49731 to 0x280000401:49760) [ 8174.557643] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8183.059632] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8186.373625] Lustre: 244257:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8196.579590] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8196.586030] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8196.593615] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8196.609604] Lustre: Skipped 4 previous similar messages [ 8196.613618] Lustre: Skipped 2 previous similar messages [ 8201.710657] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8201.716295] Lustre: Skipped 4 previous similar messages [ 8202.486073] Lustre: server umount mylustre-MDT0000 complete [ 8206.345226] LustreError: 242388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449960 with bad export cookie 9674385676634970812 [ 8206.352291] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8206.357363] LustreError: 242388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8206.680130] Lustre: server umount mylustre-MDT0001 complete [ 8220.356282] Lustre: server umount mylustre-OST0000 complete [ 8222.690428] Lustre: 235094:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788449960/real 1788449960] req@ffff97b6beae4000 x1875325461090432/t0(0) o400->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788449976 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8222.719260] Lustre: mylustre-MDT0001-lwp-OST0001: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8224.082210] Lustre: server umount mylustre-OST0001 complete [ 8243.865938] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 8253.666104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8254.229313] LustreError: 246567:0:(ldlm_lib.c:1192:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8254.321832] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8258.929970] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8269.433417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8275.010871] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8278.319155] Lustre: 247709:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8285.458540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8293.087963] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8294.168803] LustreError: 248465:0:(ldlm_lib.c:1192:target_handle_connect()) tfs-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8294.193810] LustreError: 248465:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8299.333217] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49731 to 0x280000401:49792) [ 8303.478378] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8309.250614] Lustre: tfs-OST0001: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 8310.793031] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8319.220743] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8322.629596] Lustre: 249582:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8329.702152] Lustre: tfs-MDT0000-lwp-OST0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8329.709890] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 8329.709995] Lustre: Skipped 3 previous similar messages [ 8329.734385] Lustre: Skipped 3 previous similar messages [ 8335.729978] Lustre: server umount tfs-MDT0000 complete [ 8339.341629] LustreError: 246542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450093 with bad export cookie 9674385676634972471 [ 8339.347946] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8339.354650] LustreError: 246542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8339.763891] Lustre: server umount tfs-MDT0001 complete [ 8353.357248] Lustre: server umount tfs-OST0000 complete [ 8355.300145] Lustre: 235094:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788450093/real 1788450093] req@ffff97b596afad80 x1875325461140608/t0(0) o400->tfs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788450109 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8355.333913] Lustre: tfs-MDT0001-lwp-OST0001: Connection to tfs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8357.645423] Lustre: server umount tfs-OST0001 complete [ 8383.775246] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 8396.720883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8397.304463] LustreError: 251832:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8397.337550] LustreError: 251832:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8397.420609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8397.427976] Lustre: Skipped 3 previous similar messages [ 8402.849564] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8413.145910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8418.921327] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8422.355466] Lustre: 252975:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8430.659976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8439.286886] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8442.567380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49731 to 0x280000401:49824) [ 8448.976494] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8454.652551] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 8457.465168] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8466.050331] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8470.779354] Lustre: 254845:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8478.632462] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 11:43:50 (1788450230) [ 8500.706582] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8500.719613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8500.737988] Lustre: Skipped 3 previous similar messages [ 8500.743202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8500.755265] Lustre: Skipped 7 previous similar messages [ 8502.307850] Lustre: server umount lustre-MDT0000 complete [ 8505.864172] LustreError: 251809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450259 with bad export cookie 9674385676634974144 [ 8505.878959] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8505.880348] LustreError: 251809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8512.248331] Lustre: server umount lustre-MDT0001 complete [ 8530.400452] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8530.581349] Lustre: server umount lustre-OST0000 complete [ 8541.966434] Lustre: server umount lustre-OST0001 complete [ 8551.076698] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [ 8561.886578] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 8599.654441] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 8599.654441] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 8599.654441] [ 8599.699299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 8599.916856] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8599.946175] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8600.015443] Lustre: lustre-MDT0000: new disk, initializing [ 8600.086831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8600.102116] Lustre: Skipped 3 previous similar messages [ 8600.127069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8604.740865] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8618.636514] LDISKFS-fs (dm-1): Mount option "noacl" will be removed by 3.5 [ 8618.636514] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 8618.636514] [ 8618.714453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 8619.033850] Lustre: 258031:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8619.181226] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8619.189723] Lustre: Skipped 1 previous similar message [ 8619.361578] Lustre: lustre-MDT0001: new disk, initializing [ 8619.581696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8619.586642] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8624.702576] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8637.616421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8638.094753] Lustre: lustre-OST0000: new disk, initializing [ 8638.110283] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8638.135878] Lustre: 259613:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8639.871407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8639.886338] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8640.031577] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8645.673687] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8657.126680] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8657.308290] Lustre: lustre-OST0001: new disk, initializing [ 8657.313558] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8657.323892] Lustre: 260485:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8658.774681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8658.792490] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8658.900155] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8663.808076] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8674.488243] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 11:47:06 (1788450426) [ 8690.086334] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 11:47:22 (1788450442) [ 8706.536898] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8706.542671] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8706.545984] LustreError: Skipped 1 previous similar message [ 8706.546829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8706.546845] Lustre: Skipped 2 previous similar messages [ 8706.574146] Lustre: Skipped 3 previous similar messages [ 8719.328582] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8719.454779] Lustre: server umount lustre-OST0000 complete [ 8720.374714] LustreError: 259599:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8720.398842] LustreError: 259599:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 8730.056964] Lustre: server umount lustre-MDT0000 complete [ 8735.169818] LustreError: 258826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450489 with bad export cookie 9674385676634975691 [ 8735.182567] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8735.183545] LustreError: 258826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8741.643362] Lustre: server umount lustre-MDT0001 complete [ 8753.857218] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 8826.701934] Lustre: server umount lustre-OST0001 complete [ 8834.363371] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [ 8850.495480] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 8892.008854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8892.210459] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8892.232425] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8892.342371] Lustre: lustre-MDT0000: new disk, initializing [ 8892.421839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8892.429468] Lustre: Skipped 3 previous similar messages [ 8892.446154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8897.607469] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8909.596094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8909.806086] Lustre: 265290:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8909.892897] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8909.898912] Lustre: Skipped 1 previous similar message [ 8910.027811] Lustre: lustre-MDT0001: new disk, initializing [ 8910.211635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8910.225643] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8915.086647] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8926.188244] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8931.929506] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8939.626052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8939.941851] Lustre: 267259:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8941.504920] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8941.724686] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8948.214804] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 8960.625524] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8972.257661] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8972.288330] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8972.307366] Lustre: Skipped 4 previous similar messages [ 8972.329528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8972.334630] Lustre: Skipped 10 previous similar messages [ 8974.432878] Lustre: server umount lustre-OST0000 complete [ 8981.985509] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8981.992310] LustreError: Skipped 1 previous similar message [ 8984.805630] Lustre: server umount lustre-MDT0000 complete [ 8988.164260] LustreError: 266709:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8988.195786] LustreError: 266709:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 8989.343854] LustreError: 265283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450743 with bad export cookie 9674385676634977168 [ 8989.348326] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8989.357056] LustreError: 265283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8989.847834] Lustre: server umount lustre-MDT0001 complete [ 9002.353893] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 9005.580730] Key type lgssc unregistered [ 9005.893934] LNet: 269023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9005.903838] LNetError: 269023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9005.915760] LNet: Removed LNI 192.168.204.101@tcp [ 9006.742338] Key type .llcrypt unregistered [ 9006.744048] Key type ._llcrypt unregistered [ 9033.762982] Key type ._llcrypt registered [ 9033.765937] Key type .llcrypt registered [ 9033.896866] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 9034.775093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9034.801900] alg: No test for adler32 (adler32-zlib) [ 9035.839237] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 9036.175881] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 9037.872102] Key type lgssc registered [ 9039.288135] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9047.567193] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9047.585120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9049.068898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9053.175646] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 9058.282171] LustreError: 270487:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9058.298699] LustreError: 270487:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9061.111936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9061.373117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9064.764879] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 9070.582676] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9075.562100] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9082.371645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9082.765171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9089.796646] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 9093.131996] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 9098.624531] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9099.872777] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 9102.307101] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9102.317712] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9102.356083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9103.329933] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9103.337707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9103.349904] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9103.382739] Lustre: Skipped 1 previous similar message [ 9108.430505] Lustre: server umount lustre-OST0000 complete [ 9112.546033] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9112.554978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9112.564352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9118.690539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9118.698611] Lustre: Skipped 3 previous similar messages [ 9118.996915] Lustre: server umount lustre-MDT0000 complete [ 9123.394174] LustreError: 270468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450877 with bad export cookie 13927229419115978684 [ 9123.404114] LustreError: 270468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9123.410236] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9124.003279] Lustre: server umount lustre-MDT0001 complete [ 9134.694345] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 9137.612840] Key type lgssc unregistered [ 9137.918606] LNet: 274077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9137.932721] LNetError: 274077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9137.950882] LNet: Removed LNI 192.168.204.101@tcp [ 9138.879676] Key type .llcrypt unregistered [ 9138.881467] Key type ._llcrypt unregistered [ 9157.178736] Key type ._llcrypt registered [ 9157.180900] Key type .llcrypt registered [ 9157.286706] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 9158.574357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9158.604085] alg: No test for adler32 (adler32-zlib) [ 9159.642852] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 9159.939739] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 9161.656173] Key type lgssc registered [ 9162.730567] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9172.541444] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9172.562722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9188.128993] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 9193.248401] LustreError: 275271:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.101@tcp: failed processing log, type 4: rc = -110 [ 9219.937689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9226.756886] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 9236.252312] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9238.889344] Lustre: server umount lustre-OST0000 complete [ 9259.633202] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 9262.626712] Key type lgssc unregistered [ 9262.990780] LNet: 276907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9262.995875] LNetError: 276907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9263.014435] LNet: Removed LNI 192.168.204.101@tcp [ 9263.953424] Key type .llcrypt unregistered [ 9263.960060] Key type ._llcrypt unregistered [ 9309.582915] Key type ._llcrypt registered [ 9309.586597] Key type .llcrypt registered [ 9309.713572] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 9310.939386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9311.047116] alg: No test for adler32 (adler32-zlib) [ 9312.258281] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 9312.524898] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 9314.248635] Key type lgssc registered [ 9315.438217] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9330.163447] loop0: detected capacity change from 0 to 204800000 [ 9330.383790] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9330.407641] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9330.836602] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9332.055471] loop0: detected capacity change from 0 to 204800000 [ 9332.265224] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9332.274420] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9332.671831] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9333.805872] loop0: detected capacity change from 0 to 204800000 [ 9333.999150] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9334.007573] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9334.101540] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9335.272950] loop0: detected capacity change from 0 to 204800000 [ 9335.440620] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9335.451289] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9335.561370] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9336.570562] loop0: detected capacity change from 0 to 204800000 [ 9336.599929] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9341.326814] loop0: detected capacity change from 0 to 204800000 [ 9341.364913] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9345.568839] loop0: detected capacity change from 0 to 204800000 [ 9345.599402] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9349.712031] loop0: detected capacity change from 0 to 204800000 [ 9349.748701] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9354.027503] loop0: detected capacity change from 0 to 204800000 [ 9354.103856] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9354.131293] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9354.218614] Lustre: 280325:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9354.226966] Lustre: 280325:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 9354.234413] Lustre: 280325:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 9354.243301] Lustre: 280325:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 9354.252368] Lustre: 280325:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9354.261498] Lustre: 280325:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9354.272770] Lustre: 280325:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9354.284254] Lustre: 280325:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 9354.298463] Lustre: 280325:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 9354.304683] Lustre: 280325:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 9354.312149] Lustre: 280325:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 9354.319863] Lustre: 280325:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 9354.326767] Lustre: 280325:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 9354.333402] Lustre: 280325:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 9360.943932] Lustre: server umount MGS complete [ 9361.858504] loop0: detected capacity change from 0 to 204800000 [ 9361.946479] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9362.426623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9362.488118] Lustre: lustre-MDT0000: Aborting client recovery [ 9363.457937] loop1: detected capacity change from 0 to 204800000 [ 9363.637814] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9363.776854] Lustre: 280758:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9363.796459] Lustre: 280758:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 9363.808881] Lustre: 280758:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9363.818416] Lustre: 280758:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9363.829884] Lustre: 280758:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 9363.838406] Lustre: 280758:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9363.845805] Lustre: 280758:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 9363.858186] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9364.023116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9364.068199] Lustre: lustre-MDT0001: Aborting client recovery [ 9364.976124] loop2: detected capacity change from 0 to 204800000 [ 9365.141900] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9365.256495] Lustre: 280837:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9365.274594] Lustre: 280837:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9365.288288] Lustre: 280837:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9365.296401] Lustre: 280837:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9365.303404] Lustre: 280837:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9365.310516] Lustre: 280837:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9365.317554] Lustre: 280837:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9365.407942] Lustre: Found index 0 for lustre-OST0000, updating log [ 9365.436531] Lustre: 280855:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9365.493300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9365.511499] Lustre: lustre-OST0000: Aborting recovery [ 9366.312445] loop3: detected capacity change from 0 to 204800000 [ 9366.438313] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9366.489724] Lustre: 280920:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9366.502182] Lustre: 280920:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9366.513695] Lustre: 280920:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9366.526600] Lustre: 280920:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9366.543605] Lustre: 280920:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9366.573737] Lustre: 280920:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9366.587135] Lustre: 280920:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9366.614481] Lustre: Found index 1 for lustre-OST0001, updating log [ 9366.665505] Lustre: 280925:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9366.753880] Lustre: lustre-OST0001: Aborting recovery [ 9372.147119] LustreError: 280702:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 9373.161875] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9373.231600] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9373.256465] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 9373.257763] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 9373.258300] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 9373.268380] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 9374.920664] LustreError: 280669:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 728us: rc = -2 [ 9375.421158] LustreError: 280666:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 1218us: rc = -2 [ 9375.444512] LustreError: 280666:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 19 previous similar messages [ 9388.518900] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9388.540595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9393.635286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9393.645727] Lustre: Skipped 6 previous similar messages [ 9394.881235] Lustre: server umount lustre-MDT0000 complete [ 9395.823975] LustreError: 280653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451149 with bad export cookie 4519275226366783720 [ 9395.849821] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9398.753069] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9398.754273] LustreError: 280667:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9398.754416] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9398.779276] Lustre: Skipped 4 previous similar messages [ 9398.831097] LustreError: 280667:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9400.804292] LustreError: 280933:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9400.805405] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9400.833521] Lustre: Skipped 2 previous similar messages [ 9402.331016] Lustre: server umount lustre-MDT0001 complete [ 9409.508480] Lustre: server umount lustre-OST0000 complete [ 9416.663745] Lustre: server umount lustre-OST0001 complete [ 9425.298707] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 11:59:37 (1788451177) [ 9457.218591] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [ 9471.027375] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 9511.454806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9511.698685] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9511.718425] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9511.802721] Lustre: lustre-MDT0000: new disk, initializing [ 9511.895343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9511.902542] Lustre: Skipped 1 previous similar message [ 9511.929763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9516.994830] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 9528.774399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9528.928720] Lustre: 285005:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9528.998881] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9529.004275] Lustre: Skipped 1 previous similar message [ 9529.169563] Lustre: lustre-MDT0001: new disk, initializing [ 9529.358146] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9529.417915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9529.429482] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9534.657275] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 9545.887349] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9552.161659] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9559.915232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9560.326629] Lustre: lustre-OST0000: new disk, initializing [ 9560.332346] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9560.340152] Lustre: 286974:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9560.426112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9568.038900] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 9569.859499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9569.866627] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9570.030267] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9578.811044] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9585.172362] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9595.494610] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9602.293467] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9647.089289] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9647.095287] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9647.103210] LustreError: Skipped 1 previous similar message [ 9647.105219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9647.146355] Lustre: Skipped 1 previous similar message [ 9649.582029] Lustre: server umount lustre-OST0000 complete [ 9657.316828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9657.320306] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9657.344789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9657.360201] Lustre: Skipped 1 previous similar message [ 9657.375345] Lustre: Skipped 2 previous similar messages [ 9660.500909] Lustre: server umount lustre-MDT0000 complete [ 9662.434669] LustreError: 286190:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9662.451307] LustreError: 286190:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9664.490727] LustreError: 284997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451418 with bad export cookie 4519275226366786429 [ 9664.493626] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9664.503353] LustreError: 284997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9664.897908] Lustre: server umount lustre-MDT0001 complete [ 9673.255125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9678.039799] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 9684.746996] Lustre: server umount MGS complete [ 9698.873053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9699.466209] LustreError: 290792:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9699.499358] LustreError: 290805:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9699.584875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9704.413469] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 9704.933693] LustreError: 290793:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9714.446498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9719.662761] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 9728.047547] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9734.451702] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9742.178764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9742.531074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9742.542287] Lustre: Skipped 1 previous similar message [ 9749.289710] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 9758.721544] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9790.947272] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9790.956226] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9790.959021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9790.974671] Lustre: Skipped 1 previous similar message [ 9793.533911] Lustre: server umount lustre-OST0000 complete [ 9801.192691] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9801.227182] Lustre: Skipped 1 previous similar message [ 9804.169313] Lustre: server umount lustre-MDT0000 complete [ 9806.306719] LustreError: 290792:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9806.317462] LustreError: 290792:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9807.485823] LustreError: 293521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451561 with bad export cookie 4519275226366788270 [ 9807.493408] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9807.500652] LustreError: 293521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9807.935764] Lustre: server umount lustre-MDT0001 complete [ 9827.624408] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [ 9832.187797] Key type lgssc unregistered [ 9832.666124] LNet: 294763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9832.681948] LNetError: 294763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9833.704062] LNet: Removed LNI 192.168.204.101@tcp [ 9834.731181] Key type .llcrypt unregistered [ 9834.733548] Key type ._llcrypt unregistered [ 9874.703868] Key type ._llcrypt registered [ 9874.716662] Key type .llcrypt registered [ 9874.900332] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [ 9892.172598] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [ 9893.208341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9893.268480] alg: No test for adler32 (adler32-zlib) [ 9894.555917] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 9894.870936] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [ 9896.586822] Key type lgssc registered [ 9897.887731] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9943.377601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9943.401865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9944.828235] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9944.854941] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9944.937433] Lustre: lustre-MDT0000: new disk, initializing [ 9945.063471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9945.095656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9950.713328] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 9963.193383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9963.388085] Lustre: 298725:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9963.467964] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9963.473158] Lustre: Skipped 1 previous similar message [ 9963.610106] Lustre: lustre-MDT0001: new disk, initializing [ 9963.729907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9963.771021] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9963.787299] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9969.350029] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [ 9980.527870] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9987.773987] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9995.621820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9996.110620] Lustre: lustre-OST0000: new disk, initializing [ 9996.117453] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9996.123262] Lustre: 300696:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9996.203578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10003.535476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10003.550554] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10003.720484] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10004.166237] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [10017.612804] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10026.221080] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [10040.414723] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [10042.846548] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [10052.090075] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [10096.107223] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10096.131066] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10096.145761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10096.167162] Lustre: Skipped 1 previous similar message [10096.199633] Lustre: Skipped 1 previous similar message [10099.626636] Lustre: server umount lustre-OST0000 complete [10106.337440] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10106.347419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10106.353657] Lustre: Skipped 1 previous similar message [10110.845527] Lustre: server umount lustre-MDT0000 complete [10115.069207] LustreError: 298718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451869 with bad export cookie 12878154483205316243 [10115.075081] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10115.084869] LustreError: 298718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10115.424217] Lustre: server umount lustre-MDT0001 complete [10125.183932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10130.546519] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [10136.013109] Lustre: server umount MGS complete [10149.122480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10149.536750] LustreError: 304520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10149.555061] LustreError: 304520:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10149.559880] LustreError: 304533:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [10149.648588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10154.622769] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [10154.983668] LustreError: 304521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10159.083733] LustreError: 304520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10163.699116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10164.040025] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10168.070470] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [10174.962259] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10181.851469] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10191.304732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10191.993272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10200.157904] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [10209.333594] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10244.582362] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10244.605684] LustreError: Skipped 1 previous similar message [10244.612621] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10244.636414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10244.659661] Lustre: Skipped 1 previous similar message [10244.699768] Lustre: Skipped 2 previous similar messages [10249.720723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10249.730473] Lustre: Skipped 1 previous similar message [10254.822866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10254.840596] Lustre: Skipped 1 previous similar message [10257.376191] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10257.515126] Lustre: server umount lustre-OST0000 complete [10265.072125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10265.076288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10265.103226] Lustre: Skipped 1 previous similar message [10265.114387] Lustre: Skipped 1 previous similar message [10276.320130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10276.517421] Lustre: server umount lustre-MDT0000 complete [10280.264987] LustreError: 304501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452034 with bad export cookie 12878154483205318182 [10280.267501] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10280.281096] LustreError: 304501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10280.673897] Lustre: server umount lustre-MDT0001 complete [10292.208386] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [10295.467982] Key type lgssc unregistered [10295.904227] LNet: 308474:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10295.919046] LNetError: 308474:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10297.001301] LNet: Removed LNI 192.168.204.101@tcp [10298.057465] Key type .llcrypt unregistered [10298.061168] Key type ._llcrypt unregistered [10320.593543] Key type ._llcrypt registered [10320.596343] Key type .llcrypt registered [10320.686176] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [10338.779775] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [10340.135862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10340.281064] alg: No test for adler32 (adler32-zlib) [10341.585528] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [10341.885748] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [10343.616157] Key type lgssc registered [10345.167411] Lustre: Echo OBD driver; http://www.lustre.org/ [10371.827940] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [10387.366900] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10387.395613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10388.701615] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10388.766430] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10388.932885] Lustre: lustre-MDT0000: new disk, initializing [10389.071553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10389.099974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10394.278242] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [10407.567131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10408.067952] Lustre: lustre-OST0000: new disk, initializing [10408.075905] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10408.085940] Lustre: Skipped 1 previous similar message [10408.098884] Lustre: 312526:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10408.334448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10415.817824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10415.826481] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10415.893970] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10416.170830] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [11447.384767] Lustre: server umount lustre-MDT0000 complete [11452.700571] LustreError: 311712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453206 with bad export cookie 10347677471623046651 [11452.721100] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11462.358302] Lustre: server umount lustre-OST0000 complete [11496.474233] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [11509.996233] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [11553.726384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11554.026198] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11554.072562] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11554.190737] Lustre: lustre-MDT0000: new disk, initializing [11554.375894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11554.394950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11560.217818] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [11572.263539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11572.409493] Lustre: 316894:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11572.493891] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11572.502474] Lustre: Skipped 1 previous similar message [11572.571704] Lustre: lustre-MDT0001: new disk, initializing [11572.750602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11572.789050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11572.798600] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11578.598740] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [11589.683500] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11597.053547] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11604.868149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11605.233773] Lustre: lustre-OST0000: new disk, initializing [11605.237989] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11605.246748] Lustre: 318865:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11605.410784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11612.282619] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [11613.275959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11613.283066] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11613.521508] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11622.860052] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11629.712221] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11630.109026] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11636.859744] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11637.246974] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11644.421700] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11644.426526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11644.431396] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11644.434490] Lustre: Skipped 1 previous similar message [11646.481326] Lustre: server umount lustre-OST0000 complete [11654.631736] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11654.636956] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11654.656436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11654.673275] LustreError: Skipped 1 previous similar message [11654.684603] Lustre: Skipped 2 previous similar messages [11657.687533] Lustre: server umount lustre-MDT0000 complete [11662.786848] LustreError: 316886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453416 with bad export cookie 10347677471625987470 [11662.807093] LustreError: 316886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11662.821902] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11663.253400] Lustre: server umount lustre-MDT0001 complete [11667.498662] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [11669.470733] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 12:37:01 (1788453421) [11677.723876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11678.208068] LustreError: 321178:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11678.221674] LustreError: 321178:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11678.348559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11683.011870] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [11683.819607] LustreError: 321179:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11688.930401] LustreError: 321178:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11694.051971] LustreError: 321179:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11694.546253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11695.198234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11701.914239] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [11711.462767] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11718.879437] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11727.164695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11727.489559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11734.931066] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [11744.379989] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11753.704200] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11754.012401] Lustre: lustre-OST0001: new disk, initializing [11754.016739] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11754.030364] Lustre: 324017:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11754.155865] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11762.617948] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [11763.884448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11763.903818] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11763.982646] LustreError: 324419:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11764.001766] LustreError: 324419:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [11765.045001] LustreError: 324410:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11765.066765] LustreError: 324410:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [11766.127349] LustreError: 324419:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11768.235944] LustreError: 324410:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11768.261633] LustreError: 324410:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [11772.388318] LustreError: 324419:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11772.405109] LustreError: 324419:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 8 previous similar messages [11775.797749] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11776.057797] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11780.587466] LustreError: 324410:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11780.600911] LustreError: 324410:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 14 previous similar messages [11783.390897] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff96fb6046b800.ost_server_uuid 50 [11785.334431] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff96fb6046b800.ost_server_uuid in FULL state after 0 sec [11792.695594] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff96fb6046b800.ost_server_uuid 50 [11794.508437] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff96fb6046b800.ost_server_uuid in IDLE state after 0 sec [11801.090643] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11810.277935] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11810.287244] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11810.290188] Lustre: Skipped 2 previous similar messages [11810.301927] Lustre: Skipped 1 previous similar message [11815.400825] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11815.413017] Lustre: Skipped 2 previous similar messages [11816.519755] Lustre: server umount lustre-OST0001 complete [11820.546198] LustreError: 323001:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11820.582536] LustreError: 323001:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11825.642624] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11825.645220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11825.656950] Lustre: Skipped 1 previous similar message [11825.669173] Lustre: Skipped 2 previous similar messages [11829.994850] Lustre: server umount lustre-OST0000 complete [11835.873339] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11835.877576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11835.887671] Lustre: Skipped 1 previous similar message [11835.895594] Lustre: Skipped 1 previous similar message [11841.569459] Lustre: server umount lustre-MDT0000 complete [11845.093204] LustreError: 321175:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11845.129251] LustreError: 321175:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11845.303577] LustreError: 321159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453599 with bad export cookie 10347677471625988345 [11845.307105] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11845.311433] LustreError: 321159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11845.628142] Lustre: server umount lustre-MDT0001 complete [11866.427689] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [11872.246781] Key type lgssc unregistered [11872.960293] LNet: 326733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11872.980818] LNetError: 326733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11873.002236] LNet: Removed LNI 192.168.204.101@tcp [11874.629262] Key type .llcrypt unregistered [11874.630831] Key type ._llcrypt unregistered [11907.275803] Key type ._llcrypt registered [11907.284624] Key type .llcrypt registered [11907.492889] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [11908.856948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11909.024286] alg: No test for adler32 (adler32-zlib) [11910.208638] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11910.588996] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [11912.408284] Key type lgssc registered [11914.067232] Lustre: Echo OBD driver; http://www.lustre.org/ [11922.891550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11922.927796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11924.550366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11929.667065] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [11934.691799] LustreError: 328156:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11934.715721] LustreError: 328156:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11938.617482] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11939.821955] LustreError: 328155:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11944.937408] LustreError: 328156:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11947.608956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11948.259133] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11953.835317] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [11966.170657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11966.860435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11969.964720] LustreError: 329838:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11969.986772] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [11974.058428] LustreError: 329838:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11974.085850] LustreError: 329838:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11974.182347] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [11982.244569] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11984.361251] LustreError: 329838:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11984.388249] LustreError: 329838:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11990.001312] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11990.500272] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11995.677347] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [11997.856718] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [12005.504559] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12013.015993] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [12013.403514] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12118.504369] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12118.506861] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12118.511226] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12118.519118] LustreError: Skipped 1 previous similar message [12118.546154] Lustre: Skipped 1 previous similar message [12122.610992] Lustre: server umount lustre-OST0001 complete [12123.632949] LustreError: 331083:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12123.662354] LustreError: 331083:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12128.747791] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12128.757321] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12128.771430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12128.779644] Lustre: Skipped 2 previous similar messages [12133.863433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12133.873058] Lustre: Skipped 1 previous similar message [12134.759468] Lustre: server umount lustre-OST0000 complete [12142.567730] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12142.570370] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12142.590470] Lustre: Skipped 1 previous similar message [12142.600836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12145.911730] Lustre: server umount lustre-MDT0000 complete [12150.486228] LustreError: 329090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453904 with bad export cookie 1474973777188716219 [12150.490220] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12150.499874] LustreError: 329090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12150.926326] Lustre: server umount lustre-MDT0001 complete [12162.481775] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [12166.752946] Key type lgssc unregistered [12167.231187] LNet: 333108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12167.241189] LNetError: 333108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12167.264721] LNet: Removed LNI 192.168.204.101@tcp [12168.377553] Key type .llcrypt unregistered [12168.379392] Key type ._llcrypt unregistered [12219.853795] Key type ._llcrypt registered [12219.856155] Key type .llcrypt registered [12220.026349] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [12221.534400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12221.642131] alg: No test for adler32 (adler32-zlib) [12222.936422] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12223.281301] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [12225.072474] Key type lgssc registered [12226.619666] Lustre: Echo OBD driver; http://www.lustre.org/ [12238.654132] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12238.682839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12240.253102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12246.971462] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [12250.599848] LustreError: 335396:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12250.656966] LustreError: 335396:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12255.717584] LustreError: 335397:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12259.230266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12259.668796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12265.219249] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [12268.808280] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12277.301811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12277.816957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12285.505446] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [12286.003025] LustreError: 336890:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12286.047940] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [12286.048200] LustreError: 336890:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12291.059948] LustreError: 336890:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12291.099121] LustreError: 336890:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12296.164795] LustreError: 337278:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12296.192320] LustreError: 337278:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12296.750980] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12297.135338] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12302.383645] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [12305.953321] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [12319.032883] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12327.894742] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [12346.854518] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12346.866429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12346.875122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12348.387111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12348.392256] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12348.414330] Lustre: Skipped 1 previous similar message [12351.615226] Lustre: server umount lustre-MDT0000 complete [12353.510033] LustreError: 335391:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12353.535882] LustreError: 335391:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12356.201480] LustreError: 335376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454110 with bad export cookie 9938864806672133519 [12356.218125] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12356.220910] LustreError: 335376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12356.592878] Lustre: server umount lustre-MDT0001 complete [12370.973623] Lustre: server umount lustre-OST0000 complete [12375.008133] Lustre: 334660:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788454112/real 1788454112] req@ffff97b6a43e3100 x1875329917460992/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788454128 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12375.041467] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12375.056367] Lustre: Skipped 1 previous similar message [12375.721260] Lustre: server umount lustre-OST0001 complete [12397.742872] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [12410.931382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12411.397768] LustreError: 340667:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12411.494743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12416.303450] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [12426.435387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12426.829293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12432.092799] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [12435.717893] Lustre: 341810:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [12443.876211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12444.440637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12451.638782] LustreError: 342163:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12451.670964] LustreError: 342163:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12452.127520] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [12456.769106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [12461.882257] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12467.222076] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [12469.093628] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [12477.494602] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12481.970309] Lustre: 343683:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [12488.161374] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12488.172398] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12488.186083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12488.194294] Lustre: Skipped 3 previous similar messages [12492.791478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12492.794887] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12492.810320] Lustre: Skipped 1 previous similar message [12492.838543] Lustre: Skipped 1 previous similar message [12494.639940] Lustre: server umount lustre-MDT0000 complete [12499.809387] LustreError: 340649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454253 with bad export cookie 9938864806672135220 [12499.815837] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12499.831763] LustreError: 340649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12500.349192] Lustre: server umount lustre-MDT0001 complete [12515.062948] Lustre: server umount lustre-OST0000 complete [12518.369283] Lustre: 334659:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788454256/real 1788454256] req@ffff97b6856e5c00 x1875329917511808/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788454272 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12518.405466] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12518.419159] Lustre: Skipped 1 previous similar message [12519.051742] Lustre: server umount lustre-OST0001 complete [12523.013868] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [12525.379617] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [12527.747416] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 12:51:19 (1788454279) [12555.884466] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [12566.695945] loop0: detected capacity change from 0 to 1099511627776 [12567.361295] SGI XFS with ACLs, security attributes, quota, no debug enabled [12567.366374] *** VALIDATE xfs *** [12567.375490] XFS (loop0): Mounting V5 Filesystem [12567.413621] XFS (loop0): Ending clean mount [12569.018764] loop1: detected capacity change from 0 to 18691697672192 [12689.378344] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [12700.487688] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [12740.216484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12740.558552] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12740.595546] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12740.716394] Lustre: lustre-MDT0000: new disk, initializing [12740.860564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12740.867642] Lustre: Skipped 1 previous similar message [12740.891962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12746.446086] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [12757.927467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12758.053785] Lustre: 348823:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12758.110366] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12758.114169] Lustre: Skipped 1 previous similar message [12758.185662] Lustre: lustre-MDT0001: new disk, initializing [12758.385135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12758.399476] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12763.782536] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [12776.300946] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12783.692670] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12792.224345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12792.686262] Lustre: lustre-OST0000: new disk, initializing [12792.695579] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12792.704303] Lustre: 350794:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12793.844336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12793.857066] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12793.974571] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12799.967556] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [12810.871038] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12816.174842] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12816.412807] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12822.214222] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12822.524466] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12825.057161] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12825.076440] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12825.086992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12825.093668] Lustre: Skipped 2 previous similar messages [12829.678565] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12834.787786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12834.801079] Lustre: Skipped 2 previous similar messages [12839.393211] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12839.520288] Lustre: server umount lustre-OST0000 complete [12850.599030] Lustre: server umount lustre-MDT0000 complete [12855.119430] LustreError: 348815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454609 with bad export cookie 9938864806672136774 [12855.131125] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12855.133579] LustreError: 348815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12855.551142] Lustre: server umount lustre-MDT0001 complete [12866.737126] XFS (loop0): Unmounting Filesystem [12872.420438] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 12:57:04 (1788454624) [12879.611501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12879.958546] LustreError: 353354:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12879.982728] LustreError: 353354:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [12880.045057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12880.049147] Lustre: Skipped 2 previous similar messages [12884.629546] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [12896.128544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12902.041658] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [12911.637556] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12919.617070] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12928.088176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12937.379450] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [12947.121285] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12965.345549] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12965.364314] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12965.378898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12965.400042] Lustre: Skipped 4 previous similar messages [12965.424143] Lustre: Skipped 6 previous similar messages [12968.697510] Lustre: server umount lustre-OST0000 complete [12988.384199] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12988.726696] Lustre: server umount lustre-MDT0000 complete [12993.576742] LustreError: 353335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454747 with bad export cookie 9938864806672137649 [12993.579483] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12993.591709] LustreError: 353335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12993.948947] Lustre: server umount lustre-MDT0001 complete [13007.054400] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [13010.290096] Key type lgssc unregistered [13010.606846] LNet: 356994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13010.622950] LNetError: 356994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13010.661907] LNet: Removed LNI 192.168.204.101@tcp [13011.893208] Key type .llcrypt unregistered [13011.899517] Key type ._llcrypt unregistered [13044.893646] Key type ._llcrypt registered [13044.895784] Key type .llcrypt registered [13045.032539] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [13046.454448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13046.536658] alg: No test for adler32 (adler32-zlib) [13047.949690] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [13048.290294] LNet: Added LNI 192.168.204.101@tcp [8/256/0/180] [13049.992876] Key type lgssc registered [13051.254790] Lustre: Echo OBD driver; http://www.lustre.org/ [13059.586791] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13059.609307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13061.213728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13065.744518] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [13071.340122] LustreError: 358559:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13071.361555] LustreError: 358559:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13075.307775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13075.587005] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13080.869108] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [13089.581983] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13096.474880] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13104.364321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13104.718041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13110.246326] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [13119.285415] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13130.731218] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13130.747190] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13130.760913] Lustre: Skipped 1 previous similar message [13131.857898] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.1@tcp (stopping) [13135.689088] Lustre: server umount lustre-MDT0001 complete [13135.846391] LustreError: 358559:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13135.859572] LustreError: 358559:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13136.865931] LustreError: 361086:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13140.963323] LustreError: 358560:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13140.981364] LustreError: 358560:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13146.112849] LustreError: 360029:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13146.147179] LustreError: 360029:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13148.864991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13148.931913] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13148.936703] Lustre: Found index 1 for lustre-MDT0001, updating log [13148.990433] Lustre: 358546:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13149.250768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13153.684942] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [13154.287651] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13156.382298] LustreError: 362171:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13156.412413] LustreError: 362171:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13156.429580] LustreError: 358551:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13156.561611] LustreError: 362176:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.101@tcp: cfg command failed: rc = -17 [13156.586663] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [13156.586663] [13461.476060] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13461.482245] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13461.485930] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13461.516459] Lustre: Skipped 2 previous similar messages [13464.531994] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.1@tcp (stopping) [13464.545256] Lustre: Skipped 1 previous similar message [13469.636953] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.1@tcp (stopping) [13469.648608] Lustre: Skipped 2 previous similar messages [13474.272301] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13474.546583] Lustre: server umount lustre-MDT0001 complete [13474.771471] LustreError: 358555:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13474.795487] LustreError: 358555:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13492.197302] LustreError: 358559:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13492.247941] LustreError: 358559:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [13492.928427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13493.504875] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13493.515864] Lustre: Found index 1 for lustre-MDT0001, updating log [13493.608268] Lustre: 358546:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13494.131800] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13499.398898] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13503.500701] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [13503.532291] LustreError: 370399:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13503.549254] LustreError: 370399:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13503.567523] LustreError: 358551:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13503.577741] LustreError: 358551:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [13503.730528] LustreError: 370406:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [13503.745380] LustreError: 370406:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [13503.758756] LustreError: 370406:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.101@tcp: cfg command failed: rc = -17 [13503.776276] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.204.101@tcp [13503.776276] [13811.686905] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13811.698233] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13811.721289] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13811.739581] Lustre: Skipped 3 previous similar messages [13816.070122] Lustre: server umount lustre-MDT0001 complete [13816.806925] LustreError: 358555:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13816.833290] LustreError: 358555:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13830.457708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13830.556804] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13830.563270] Lustre: Found index 1 for lustre-MDT0001, updating log [13830.624914] Lustre: 358547:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13830.895410] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13836.276981] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13836.577840] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [13837.812311] LustreError: 378538:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13837.816891] LustreError: 378538:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13837.835213] LustreError: 358551:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13837.842506] LustreError: 358551:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [13837.903889] LustreError: 378545:0:(lod_lov.c:210:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [13837.929777] LustreError: 378545:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.101@tcp: cfg command failed: rc = -114 [13837.941310] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [13837.941310] [14158.818963] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14158.820471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14158.821059] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14158.835737] Lustre: Skipped 3 previous similar messages [14158.840530] Lustre: Skipped 2 previous similar messages [14164.282840] Lustre: server umount lustre-MDT0000 complete [14167.877098] LustreError: 370136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455921 with bad export cookie 10860918497230213328 [14167.882577] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14167.887587] LustreError: 370136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14169.058647] LustreError: 358556:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14169.058912] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14169.078060] LustreError: 358556:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [14182.369387] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14182.721470] Lustre: server umount lustre-MDT0001 complete [14187.945812] Lustre: server umount lustre-OST0000 complete [14198.955949] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [14209.332429] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [14257.968578] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 13:20:09 (1788456009) [14266.663373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14266.831519] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14266.853366] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14266.983234] Lustre: lustre-MDT0000: new disk, initializing [14267.052729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14267.066550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14273.069862] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [14286.854382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14286.961813] Lustre: 388962:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14287.038955] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14287.047558] Lustre: Skipped 1 previous similar message [14287.195360] Lustre: lustre-MDT0001: new disk, initializing [14287.390187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14287.463294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14287.482256] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14291.699157] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [14300.972526] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14308.261390] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14315.767419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14316.018795] Lustre: lustre-OST0000: new disk, initializing [14316.024212] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14316.029397] Lustre: 390933:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14316.136680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14323.289765] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_default_debug -1 all [14326.308947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14326.327433] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14326.503943] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14333.665834] Lustre: DEBUG MARKER: oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14343.651583] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14343.662087] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14343.683197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14343.701283] Lustre: Skipped 8 previous similar messages [14349.548603] Lustre: server umount lustre-MDT0000 complete [14352.363503] LustreError: 388973:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14352.393813] LustreError: 388973:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [14354.943846] LustreError: 388951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788456108 with bad export cookie 10860918497230215281 [14354.955782] LustreError: MGC192.168.204.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14354.972840] LustreError: 388951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14355.470268] Lustre: server umount lustre-MDT0001 complete [14371.092815] Lustre: server umount lustre-OST0000 complete [14405.052589] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing set_hostid [14417.699391] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing load_modules_local [14456.908475] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14174 sec ========= 13:23:28 (1788456208) [14459.168584] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:23:30 (1788456210) === [14463.470489] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:23:35 (1788456215) === [14508.123600] Lustre: DEBUG MARKER: oleg401-server.virtnet: executing unload_modules_local [14511.535983] Key type lgssc unregistered [14511.974447] LNet: 397306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14511.986403] LNetError: 397306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14512.008654] LNet: Removed LNI 192.168.204.101@tcp [14512.918620] Key type .llcrypt unregistered [14512.921944] Key type ._llcrypt unregistered