[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 457107504 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002164] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.006552] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007039] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008018] pid_max: default: 32768 minimum: 301 [ 0.009150] LSM: Security Framework initializing [ 0.010044] Yama: becoming mindful. [ 0.011050] SELinux: Initializing. [ 0.012079] *** VALIDATE selinux *** [ 0.020789] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026380] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027196] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028104] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030025] *** VALIDATE tmpfs *** [ 0.031433] *** VALIDATE proc *** [ 0.032254] *** VALIDATE cgroup *** [ 0.034000] *** VALIDATE cgroup2 *** [ 0.035082] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036173] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037016] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038040] Spectre V2 : User space: Vulnerable [ 0.039014] Speculative Store Bypass: Vulnerable [ 0.043058] debug: unmapping init [mem 0xffffffffa8059000-0xffffffffa8060fff] [ 0.045253] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046956] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048036] ... version: 2 [ 0.049021] ... bit width: 48 [ 0.050020] ... generic registers: 4 [ 0.051011] ... value mask: 0000ffffffffffff [ 0.052015] ... max period: 00007fffffffffff [ 0.053023] ... fixed-purpose events: 3 [ 0.054024] ... event mask: 000000070000000f [ 0.055474] rcu: Hierarchical SRCU implementation. [ 0.057618] smp: Bringing up secondary CPUs ... [ 0.058591] x86: Booting SMP configuration: [ 0.059026] .... node #0, CPUs: #1 #2 #3 [ 0.066330] smp: Brought up 1 node, 4 CPUs [ 0.068010] smpboot: Max logical packages: 1 [ 0.069011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.147023] node 0 deferred pages initialised in 76ms [ 0.150027] devtmpfs: initialized [ 0.151306] x86/mm: Memory block size: 128MB [ 0.155029] gcov: version magic: 0x41383552 [ 0.156695] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.162474] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.165681] pinctrl core: initialized pinctrl subsystem [ 0.167138] [ 0.167795] ************************************************************* [ 0.170017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.173010] ** ** [ 0.174008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.176010] ** ** [ 0.178009] ** This means that this kernel is built to expose internal ** [ 0.180009] ** IOMMU data structures, which may compromise security on ** [ 0.183011] ** your system. ** [ 0.185010] ** ** [ 0.188009] ** If you see this message and you are not debugging the ** [ 0.190008] ** kernel, report this immediately to your vendor! ** [ 0.193011] ** ** [ 0.195008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.197032] ************************************************************* [ 0.200539] NET: Registered protocol family 16 [ 0.202420] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.205047] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.208049] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.212107] cpuidle: using governor menu [ 0.214016] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.217400] PCI: Using configuration type 1 for base access [ 0.219181] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.228141] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.229017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.231031] cryptd: max_cpu_qlen set to 1000 [ 0.233209] ACPI: Added _OSI(Module Device) [ 0.235009] ACPI: Added _OSI(Processor Device) [ 0.236006] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.238010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.243000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.249480] ACPI: Interpreter enabled [ 0.250000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.252013] ACPI: Using IOAPIC for interrupt routing [ 0.254086] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.257285] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.263139] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.265024] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.268016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.271084] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.276658] acpiphp: Slot [2] registered [ 0.277066] acpiphp: Slot [5] registered [ 0.278099] acpiphp: Slot [6] registered [ 0.279055] acpiphp: Slot [7] registered [ 0.281104] acpiphp: Slot [8] registered [ 0.282133] acpiphp: Slot [9] registered [ 0.285234] acpiphp: Slot [10] registered [ 0.288134] acpiphp: Slot [3] registered [ 0.291523] acpiphp: Slot [4] registered [ 0.293155] acpiphp: Slot [11] registered [ 0.295091] acpiphp: Slot [12] registered [ 0.298227] acpiphp: Slot [13] registered [ 0.301211] acpiphp: Slot [14] registered [ 0.303084] acpiphp: Slot [15] registered [ 0.305109] acpiphp: Slot [16] registered [ 0.308146] acpiphp: Slot [17] registered [ 0.310098] acpiphp: Slot [18] registered [ 0.310851] acpiphp: Slot [19] registered [ 0.312052] acpiphp: Slot [20] registered [ 0.314073] acpiphp: Slot [21] registered [ 0.314906] acpiphp: Slot [22] registered [ 0.324074] acpiphp: Slot [23] registered [ 0.325159] acpiphp: Slot [24] registered [ 0.326063] acpiphp: Slot [25] registered [ 0.328200] acpiphp: Slot [26] registered [ 0.330217] acpiphp: Slot [27] registered [ 0.332168] acpiphp: Slot [28] registered [ 0.334414] acpiphp: Slot [29] registered [ 0.336455] acpiphp: Slot [30] registered [ 0.338477] acpiphp: Slot [31] registered [ 0.341235] PCI host bridge to bus 0000:00 [ 0.343026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.347025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.350138] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.353025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.357028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.360029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.362258] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.368201] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.374819] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.392904] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.401085] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.408170] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.410030] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.414067] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.416866] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.417711] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.420037] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.421747] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.426013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.438030] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.444022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.454142] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.465018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.470014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.483016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.491062] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.500016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.504021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.518017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.525624] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.531012] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.536015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.557016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.571029] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.585018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.592017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.617021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.634968] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.646019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.656017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.684014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.701203] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.709013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.716016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.733014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.748342] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.750370] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.753496] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.755849] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.757192] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.761042] iommu: Default domain type: Passthrough [ 0.764326] SCSI subsystem initialized [ 0.765143] ACPI: bus type USB registered [ 0.766000] usbcore: registered new interface driver usbfs [ 0.766000] usbcore: registered new interface driver hub [ 0.767105] usbcore: registered new device driver usb [ 0.768000] pps_core: LinuxPPS API ver. 1 registered [ 0.768000] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.768000] PTP clock support registered [ 0.769037] EDAC MC: Ver: 3.0.0 [ 0.771599] PCI: Using ACPI for IRQ routing [ 0.774504] NetLabel: Initializing [ 0.777021] NetLabel: domain hash size = 128 [ 0.779014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.780088] NetLabel: unlabeled traffic allowed by default [ 0.782079] vgaarb: loaded [ 0.788360] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.791018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.799607] clocksource: Switched to clocksource kvm-clock [ 0.936842] VFS: Disk quotas dquot_6.6.0 [ 0.939984] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.944110] *** VALIDATE ramfs *** [ 0.945141] *** VALIDATE hugetlbfs *** [ 0.946437] pnp: PnP ACPI init [ 0.948566] pnp: PnP ACPI: found 6 devices [ 0.964402] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.966970] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.968797] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.970576] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.972708] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.974609] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.977086] NET: Registered protocol family 2 [ 0.979154] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.983404] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.986207] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.991040] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.993697] TCP: Hash tables configured (established 65536 bind 65536) [ 0.995868] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.998104] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.000091] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.002256] NET: Registered protocol family 1 [ 1.004474] RPC: Registered named UNIX socket transport module. [ 1.005915] RPC: Registered udp transport module. [ 1.007198] RPC: Registered tcp transport module. [ 1.008258] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.009837] NET: Registered protocol family 44 [ 1.011088] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.012509] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.013841] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.015273] PCI: CLS 0 bytes, default 64 [ 1.016754] Unpacking initramfs... [ 2.443582] debug: unmapping init [mem 0xffff99d77cc54000-0xffff99d77ffbffff] [ 2.447716] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.451553] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.456524] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.356663] Initialise system trusted keyrings [ 3.361126] Key type blacklist registered [ 3.363827] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.374694] zbud: loaded [ 3.378118] *** VALIDATE nfs *** [ 3.379609] *** VALIDATE nfs4 *** [ 3.381327] pstore: using deflate compression [ 3.385938] Platform Keyring initialized [ 3.771271] NET: Registered protocol family 38 [ 3.773877] Key type asymmetric registered [ 3.775489] Asymmetric key parser 'x509' registered [ 3.777679] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.781117] io scheduler mq-deadline registered [ 3.783020] io scheduler kyber registered [ 3.784891] io scheduler bfq registered [ 4.106399] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.110403] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.113968] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.117557] ACPI: Power Button [PWRF] [ 4.123818] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.131219] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.197500] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.217424] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.231898] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.259279] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.289603] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.295414] Non-volatile memory driver v1.3 [ 4.297584] Linux agpgart interface v0.103 [ 4.335102] virtio_blk virtio1: [vda] 146656 512-byte logical blocks (75.1 MB/71.6 MiB) [ 4.337410] vda: detected capacity change from 0 to 75087872 [ 4.352484] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.355813] vdb: detected capacity change from 0 to 1073741824 [ 4.371556] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.374816] vdc: detected capacity change from 0 to 2621440000 [ 4.390535] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.393810] vdd: detected capacity change from 0 to 2621440000 [ 4.415315] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.418608] vde: detected capacity change from 0 to 4294967296 [ 4.440330] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.444224] vdf: detected capacity change from 0 to 4294967296 [ 4.452226] libphy: Fixed MDIO Bus: probed [ 4.458959] usbcore: registered new interface driver usbserial_generic [ 4.461082] usbserial: USB Serial support registered for generic [ 4.463087] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.466859] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.468420] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.471362] mousedev: PS/2 mouse device common for all mice [ 4.485434] rtc_cmos 00:05: RTC can wake from S4 [ 4.486232] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.491359] rtc_cmos 00:05: registered as rtc0 [ 4.498231] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.503119] intel_pstate: CPU model not supported [ 4.507209] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.511684] hid: raw HID events driver (C) Jiri Kosina [ 4.517338] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.519266] usbcore: registered new interface driver usbhid [ 4.519271] usbhid: USB HID core driver [ 4.519479] drop_monitor: Initializing network drop monitor service [ 4.519629] Initializing XFRM netlink socket [ 4.520166] NET: Registered protocol family 10 [ 4.528503] Segment Routing with IPv6 [ 4.539736] NET: Registered protocol family 17 [ 4.541897] mpls_gso: MPLS GSO support [ 4.549865] RAS: Correctable Errors collector initialized. [ 4.552439] AVX version of gcm_enc/dec engaged. [ 4.554625] AES CTR mode by8 optimization enabled [ 4.695633] sched_clock: Marking stable (4695612213, 0)->(5525930564, -830318351) [ 4.701621] registered taskstats version 1 [ 4.703573] Loading compiled-in X.509 certificates [ 4.707609] zswap: loaded using pool lzo/zbud [ 4.745401] Key type big_key registered [ 4.762218] Key type encrypted registered [ 4.764450] ima: No TPM chip found, activating TPM-bypass! [ 4.766988] ima: Allocated hash algorithm: sha1 [ 4.769308] ima: No architecture policies found [ 4.771562] evm: Initialising EVM extended attributes: [ 4.773858] evm: security.selinux [ 4.775516] evm: security.ima [ 4.776913] evm: security.capability [ 4.778756] evm: HMAC attrs: 0x1 [ 4.781661] rtc_cmos 00:05: setting system clock to 2026-09-08 01:17:28 UTC (1788830248) [ 4.790494] debug: unmapping init [mem 0xffffffffa9003000-0xffffffffa91fffff] [ 4.794225] debug: unmapping init [mem 0xffffffffa7d82000-0xffffffffa8058fff] [ 4.803069] Write protecting the kernel read-only data: 28672k [ 4.816083] debug: unmapping init [mem 0xffffffffa6403000-0xffffffffa65fffff] [ 4.820309] debug: unmapping init [mem 0xffffffffa6d14000-0xffffffffa6dfffff] [ 4.936937] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.948567] systemd[1]: Detected virtualization kvm. [ 4.950158] systemd[1]: Detected architecture x86-64. [ 4.952813] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.985917] systemd[1]: No hostname configured. [ 4.988112] systemd[1]: Set hostname to . [ 4.990017] random: systemd: uninitialized urandom read (16 bytes read) [ 4.991917] systemd[1]: Initializing machine ID from random generator. [ 5.100285] random: ln: uninitialized urandom read (6 bytes read) [ 5.223624] random: systemd: uninitialized urandom read (16 bytes read) [ 5.226830] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 5.239225] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 5.253558] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.517740] device-mapper: uevent: version 1.0.3 [ 6.520349] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK [ 9.131565] virtio_net virtio0 ens2: renamed from eth0 ] [ 9.261463] random: fast init done Reached target System Initialization. [ OK ] Reached target Basic System. [ 9.338553] scsi host0: ata_piix [ 9.344786] scsi host1: ata_piix [ 9.346223] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.348551] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 17.414840] random: crng init done [ 17.426922] random: 7 urandom warning(s) missed due to ratelimiting [ 21.558192] dracut-initqueue[593]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 23.177931] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 26.091578] printk: systemd: 26 output lines suppressed due to ratelimiting [ 26.558909] SELinux: Disabled at runtime. [ 26.802832] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 26.815053] systemd[1]: Detected virtualization kvm. [ 26.817817] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 28.804214] systemd[1]: initrd-switch-root.service: Succeeded. [ 28.859995] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 28.867165] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 28.873331] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 28.879874] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 28.889377] systemd[1]: Starting Journal Service... Starting Journal Service... [ 28.896918] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ 29.391093] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started udev Coldplug all Devices. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 31.447076] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 32.253776] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 32.330301] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 33.488749] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 33.783968] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 38.670400] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 39.162715] NFS: Registering the id_resolver key type [ 39.165288] Key type id_resolver registered [ 39.167111] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg248-server login: [ 62.189173] hrtimer: interrupt took 7189710 ns [ 111.871711] libcfs: loading out-of-tree module taints kernel. [ 111.917347] Key type ._llcrypt registered [ 111.919447] Key type .llcrypt registered [ 112.010771] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [ 128.825400] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 130.368084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 130.390373] alg: No test for adler32 (adler32-zlib) [ 131.822273] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 132.557435] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 134.272197] Key type lgssc registered [ 135.701857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 148.470407] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 184.999538] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 196.741921] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 196.763702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 198.054713] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 198.091302] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 198.163671] Lustre: lustre-MDT0000: new disk, initializing [ 198.223672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 198.237351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 201.725847] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 213.813351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 213.927610] Lustre: 6519:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 213.967995] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 213.972687] Lustre: Skipped 1 previous similar message [ 214.022166] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 214.088426] Lustre: lustre-MDT0001: new disk, initializing [ 214.159494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 214.198519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 214.222553] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 218.556592] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.213817] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 233.083978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 233.394607] Lustre: lustre-OST0000: new disk, initializing [ 233.399918] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 233.408114] Lustre: 8454:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 233.480298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 235.622318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 235.637333] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 235.732458] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 239.019499] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 251.540705] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 251.689524] Lustre: lustre-OST0001: new disk, initializing [ 251.692778] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 251.696530] Lustre: 9526:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 251.757875] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 256.473659] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 259.631936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 259.646366] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 259.705651] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 266.973987] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 271.896935] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 277.828641] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing check_logdir /tmp/testlogs/ [ 281.949418] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing yml_node [ 285.616957] Lustre: DEBUG MARKER: Client: 2.17.57.104 [ 287.760556] Lustre: DEBUG MARKER: MDS: 2.17.57.104 [ 289.815597] Lustre: DEBUG MARKER: OSS: 2.17.57.104 [ 291.499912] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Sep 7 21:22:13 EDT 2026 [ 306.025225] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 307.327601] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 326.112857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 326.115327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 326.140777] Lustre: Skipped 3 previous similar messages [ 326.159471] Lustre: Skipped 3 previous similar messages [ 330.853757] Lustre: server umount lustre-MDT0000 complete [ 334.617783] LustreError: 9536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788830578 with bad export cookie 8357854464954614131 [ 334.620713] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 334.631948] LustreError: 9536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 336.360405] LustreError: 6525:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 336.361401] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 336.380795] LustreError: 6525:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 336.408263] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 339.368136] LustreError: 6524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 339.369366] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 339.393684] LustreError: 6524:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 339.405808] Lustre: Skipped 2 previous similar messages [ 341.033656] Lustre: server umount lustre-MDT0001 complete [ 344.217388] Lustre: server umount lustre-OST0000 complete [ 347.891368] Lustre: server umount lustre-OST0001 complete [ 353.689140] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [ 362.177866] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 392.480442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 392.731337] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 392.770867] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 392.891848] Lustre: lustre-MDT0000: new disk, initializing [ 392.962080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 392.974487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 396.794282] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 406.349931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 406.503314] Lustre: 15073:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 406.559972] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 406.564906] Lustre: Skipped 1 previous similar message [ 406.650979] Lustre: lustre-MDT0001: new disk, initializing [ 406.742439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 406.786982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 406.799213] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 410.425483] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 417.897146] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 422.199418] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 427.410327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 427.612792] Lustre: lustre-OST0000: new disk, initializing [ 427.615975] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 427.621610] Lustre: 17044:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 427.701803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 428.973686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 429.000298] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 429.072640] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 432.917931] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 441.417291] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 445.132565] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 445.311517] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 448.558814] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 448.707564] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 454.628160] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 454.635056] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 454.635923] LustreError: Skipped 1 previous similar message [ 454.645963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 454.648817] Lustre: Skipped 2 previous similar messages [ 454.659588] Lustre: Skipped 1 previous similar message [ 456.462577] Lustre: server umount lustre-OST0000 complete [ 459.746160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 459.770597] Lustre: Skipped 1 previous similar message [ 464.865863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 464.870962] Lustre: Skipped 3 previous similar messages [ 465.742595] Lustre: server umount lustre-MDT0000 complete [ 469.085747] LustreError: 15065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788830712 with bad export cookie 8357854464954615664 [ 469.097222] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 469.100218] LustreError: 15065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 469.391107] Lustre: server umount lustre-MDT0001 complete [ 473.538489] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 21:25:15 (1788830715) [ 479.477210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 479.833338] LustreError: 19305:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 479.900858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 484.364576] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 485.354331] LustreError: 19306:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 489.453149] LustreError: 19305:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 491.440373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 495.553475] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 503.019471] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 507.681317] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 513.330584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 518.602630] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 525.356697] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 528.633839] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 535.526183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 535.535079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 535.544788] Lustre: Skipped 2 previous similar messages [ 535.552109] Lustre: Skipped 2 previous similar messages [ 540.135451] Lustre: server umount lustre-MDT0000 complete [ 540.651047] LustreError: 19305:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 540.664702] LustreError: 19305:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 543.163990] LustreError: 19286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788830786 with bad export cookie 8357854464954616546 [ 543.166392] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 543.172389] LustreError: 19286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 543.421865] Lustre: server umount lustre-MDT0001 complete [ 557.122954] Lustre: server umount lustre-OST0000 complete [ 575.700311] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 589.972719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 591.243955] LustreError: 23685:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 591.353320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 591.367341] Lustre: Skipped 2 previous similar messages [ 597.914597] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 611.180926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 619.113649] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 624.295750] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 635.118444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 645.474732] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 656.689174] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 656.828666] Lustre: lustre-OST0001: new disk, initializing [ 656.835699] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 656.840068] Lustre: 26004:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 658.602935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 658.616638] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 658.805354] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 665.067505] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 674.615909] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 678.190523] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 693.729469] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 693.742362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 693.757605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 697.459369] Lustre: server umount lustre-MDT0000 complete [ 699.881651] LustreError: 23685:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 699.935078] LustreError: 23685:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 701.352077] LustreError: 23665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788830945 with bad export cookie 8357854464954617393 [ 701.356457] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 701.367861] LustreError: 23665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 701.900770] Lustre: server umount lustre-MDT0001 complete [ 715.832727] Lustre: server umount lustre-OST0000 complete [ 729.711407] Lustre: server umount lustre-OST0001 complete [ 746.019392] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 756.226934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 756.726326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 756.737718] Lustre: Skipped 3 previous similar messages [ 761.710499] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 766.951644] LustreError: 29026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 766.961857] LustreError: 29026:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 770.725356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 775.696441] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 778.764871] Lustre: 30174:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 786.202345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 794.501522] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 803.313944] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 811.342618] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 820.293386] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 824.216254] Lustre: 32050:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 832.499340] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 832.507520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 832.522910] Lustre: Skipped 3 previous similar messages [ 832.535076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 832.541826] Lustre: Skipped 4 previous similar messages [ 837.562318] Lustre: server umount lustre-MDT0000 complete [ 841.131403] LustreError: 29007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788831084 with bad export cookie 8357854464954618688 [ 841.134817] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 841.144069] LustreError: 29007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 841.405460] Lustre: server umount lustre-MDT0001 complete [ 855.500873] Lustre: server umount lustre-OST0000 complete [ 860.645286] Lustre: 3654:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788831088/real 1788831088] req@ffff99d6c2498e00 x1875724600630912/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788831104 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 860.917804] Lustre: server umount lustre-OST0001 complete [ 872.770685] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 21:31:53 (1788831113) [ 903.697404] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 915.660543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 916.279818] LustreError: 34855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 916.320280] LustreError: 34855:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 921.555435] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 930.236604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 935.862701] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 939.410484] Lustre: 36004:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 947.700312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 955.530848] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 965.728068] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 974.342795] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 983.035478] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 986.907391] Lustre: 37884:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 996.839719] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 996.843060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 996.862744] Lustre: Skipped 7 previous similar messages [ 996.880790] Lustre: Skipped 7 previous similar messages [ 999.300373] Lustre: server umount lustre-MDT0000 complete [ 1002.998923] LustreError: 34836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788831246 with bad export cookie 8357854464954619794 [ 1003.000615] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1003.025030] LustreError: 34836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1003.385835] Lustre: server umount lustre-MDT0001 complete [ 1018.428351] Lustre: server umount lustre-OST0000 complete [ 1032.833564] Lustre: server umount lustre-OST0001 complete [ 1043.055325] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 21:34:44 (1788831284) [ 1059.775269] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 1071.602327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1072.336992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1072.342980] Lustre: Skipped 7 previous similar messages [ 1077.551796] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1087.254755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1092.824733] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1096.389765] Lustre: 41182:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1103.743918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1110.836553] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1121.365766] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1128.562224] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1136.651897] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1140.766609] Lustre: 43059:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1142.091421] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1142.095639] Lustre: Skipped 1 previous similar message [ 1147.370023] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1147.387863] Lustre: Skipped 1 previous similar message [ 1152.032566] Lustre: server umount lustre-MDT0000 complete [ 1156.286423] LustreError: 40013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788831400 with bad export cookie 8357854464954620900 [ 1156.288338] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1156.304832] LustreError: 40013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1162.832377] Lustre: server umount lustre-MDT0001 complete [ 1171.227979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1182.182699] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x73fd105aa408604b [ 1182.201499] Lustre: MGC192.168.202.148@tcp: Connection restored to 0@lo (at 0@lo) [ 1182.566599] LustreError: 41546:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1182.599593] LustreError: 41546:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 27 previous similar messages [ 1187.328111] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1191.929061] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1196.679045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1202.697539] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1202.707603] Lustre: Skipped 1 previous similar message [ 1203.108450] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1213.273134] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1215.181274] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1222.192336] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1223.944889] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1232.866047] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1237.284464] Lustre: server umount lustre-MDT0000 complete [ 1241.494694] LustreError: 40011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788831485 with bad export cookie 8357854464954622027 [ 1241.507146] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1241.520386] LustreError: 40011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1242.167720] Lustre: server umount lustre-MDT0001 complete [ 1255.720788] Lustre: server umount lustre-OST0000 complete [ 1268.722355] Lustre: server umount lustre-OST0001 complete [ 1277.032850] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 21:38:39 (1788831519) [ 1295.205481] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 1306.642774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1313.035476] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1323.780673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1329.054523] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1332.514306] Lustre: 48824:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1341.555061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1349.383983] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1357.626227] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1364.483995] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1373.968276] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1380.039838] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1396.195382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1396.209368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1396.230991] Lustre: Skipped 6 previous similar messages [ 1396.244345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1396.248216] Lustre: Skipped 17 previous similar messages [ 1401.198551] Lustre: server umount lustre-MDT0000 complete [ 1404.940848] LustreError: 47652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788831648 with bad export cookie 8357854464954622818 [ 1404.943112] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1404.969143] LustreError: 47652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1405.675422] Lustre: server umount lustre-MDT0001 complete [ 1420.671403] Lustre: server umount lustre-OST0000 complete [ 1424.352111] Lustre: 3655:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788831652/real 1788831652] req@ffff99d7f9bc1c00 x1875724600835328/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788831668 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1424.951349] Lustre: server umount lustre-OST0001 complete [ 1436.446890] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 21:41:17 (1788831677) [ 1440.000834] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1442.851822] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 21:41:23 (1788831683) [ 1453.842763] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [ 1465.821723] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 1492.440571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1492.850668] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1492.945001] Lustre: lustre-MDT0000: new disk, initializing [ 1493.070335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1497.216297] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1507.366992] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1515.350613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1515.812866] Lustre: lustre-OST0000: new disk, initializing [ 1515.828466] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1515.839663] Lustre: Skipped 1 previous similar message [ 1515.853330] Lustre: 54905:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1517.038823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1517.047238] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1517.077876] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1522.871221] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1532.256406] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1534.615092] Lustre: DEBUG MARKER: create test files [ 1566.471601] Lustre: server umount lustre-OST0000 complete [ 1580.706926] Lustre: server umount lustre-MDT0000 complete [ 1591.204128] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 1594.385513] Key type lgssc unregistered [ 1594.707719] LNet: 56535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1594.714182] LNetError: 56535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1594.735162] LNet: Removed LNI 192.168.202.148@tcp [ 1595.784132] Key type .llcrypt unregistered [ 1595.786049] Key type ._llcrypt unregistered [ 1627.022973] Key type ._llcrypt registered [ 1627.024343] Key type .llcrypt registered [ 1627.160886] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 1628.281812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1628.311205] alg: No test for adler32 (adler32-zlib) [ 1629.456882] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 1629.713593] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 1631.456945] Key type lgssc registered [ 1632.584690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1640.235214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1640.260688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1641.566199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1645.334989] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1652.401934] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1658.374107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1658.744278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1665.555196] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1666.946795] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1667.094311] Lustre: 59265:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 511 < left 544, rollback = 9 [ 1667.114885] Lustre: 59265:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/4, destroy: 0/0/0 [ 1667.129761] Lustre: 59265:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1667.137715] Lustre: 59265:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1667.162660] Lustre: 59265:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1667.174671] Lustre: 59265:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1673.132995] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1678.777844] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1684.297541] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1691.630712] Lustre: server umount lustre-OST0000 complete [ 1706.190212] Lustre: server umount lustre-MDT0000 complete [ 1717.841807] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 1720.551645] Key type lgssc unregistered [ 1720.897379] LNet: 60869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1720.910691] LNetError: 60869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1720.948970] LNet: Removed LNI 192.168.202.148@tcp [ 1721.822175] Key type .llcrypt unregistered [ 1721.823746] Key type ._llcrypt unregistered [ 1754.477390] Key type ._llcrypt registered [ 1754.479823] Key type .llcrypt registered [ 1754.627191] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 1756.299608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1756.418409] alg: No test for adler32 (adler32-zlib) [ 1757.518376] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 1757.724261] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 1759.392249] Key type lgssc registered [ 1760.812514] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1767.821373] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1767.841783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1769.258177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1773.967642] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1781.337448] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1787.783266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1788.137126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1794.293802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1794.378156] Lustre: 63703:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1794.386146] Lustre: 63703:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1794.393639] Lustre: 63703:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1794.398469] Lustre: 63703:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1794.404187] Lustre: 63703:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1794.410839] Lustre: 63703:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1794.417404] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1801.550659] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1807.434257] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1815.011498] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1815.043118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1820.129262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1820.675526] Lustre: server umount lustre-OST0000 complete [ 1823.999697] Lustre: server umount lustre-MDT0000 complete [ 1832.751281] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 1834.712297] Key type lgssc unregistered [ 1834.962603] LNet: 65256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1834.967960] LNetError: 65256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1836.011114] LNet: Removed LNI 192.168.202.148@tcp [ 1836.862912] Key type .llcrypt unregistered [ 1836.865084] Key type ._llcrypt unregistered [ 1854.039728] Key type ._llcrypt registered [ 1854.044410] Key type .llcrypt registered [ 1854.141594] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [ 1869.172169] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 1870.207263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1870.475322] alg: No test for adler32 (adler32-zlib) [ 1871.725575] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 1871.911698] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 1873.568204] Key type lgssc registered [ 1874.838973] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1909.739795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1909.758141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1911.106030] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1911.153492] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1911.274372] Lustre: lustre-MDT0000: new disk, initializing [ 1911.371645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1911.397739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1915.716521] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1927.209330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1927.555357] Lustre: 68403:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1927.644883] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1927.647753] Lustre: Skipped 1 previous similar message [ 1927.758465] Lustre: lustre-MDT0001: new disk, initializing [ 1927.852897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1927.897951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1927.921532] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1932.596286] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1943.006128] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1948.912302] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1956.076653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1956.358688] Lustre: lustre-OST0000: new disk, initializing [ 1956.361929] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1956.367189] Lustre: 70373:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1956.483780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1963.276365] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1964.630573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1964.637959] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1964.819078] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1973.322533] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1978.466080] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1978.689078] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1983.824804] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1984.044514] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1990.641024] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1990.649159] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1990.665074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1990.674302] Lustre: Skipped 1 previous similar message [ 1990.686871] Lustre: Skipped 1 previous similar message [ 1995.752905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1995.759805] Lustre: Skipped 1 previous similar message [ 2000.353928] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2000.614498] Lustre: server umount lustre-OST0000 complete [ 2004.962943] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2004.967667] LustreError: Skipped 1 previous similar message [ 2004.971795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2004.981536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2006.007499] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2011.104955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2011.106874] Lustre: Skipped 2 previous similar messages [ 2016.234406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2016.238793] Lustre: Skipped 1 previous similar message [ 2019.299689] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2019.607116] Lustre: server umount lustre-MDT0000 complete [ 2024.507554] LustreError: 69198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788832268 with bad export cookie 2224209514962654091 [ 2024.516707] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2024.527895] LustreError: 69198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2024.923385] Lustre: server umount lustre-MDT0001 complete [ 2036.472720] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 21:51:17 (1788832277) [ 2051.728543] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 2094.603339] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [ 2105.513638] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 2143.342158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2143.658497] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2143.697129] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2143.788333] Lustre: lustre-MDT0000: new disk, initializing [ 2143.875808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2143.886152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2148.192130] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2159.758278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2159.877448] Lustre: 76416:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2159.926822] LustreError: 76429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2159.943758] LustreError: 76429:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2159.965411] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2159.968623] Lustre: Skipped 1 previous similar message [ 2160.067717] Lustre: lustre-MDT0001: new disk, initializing [ 2160.216866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2160.293423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2160.308971] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2164.651258] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2173.871977] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2179.567171] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2186.094382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2186.447697] Lustre: lustre-OST0000: new disk, initializing [ 2186.450873] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2186.456142] Lustre: 78389:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2186.536838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2193.157612] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2195.034454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2195.050681] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2195.228303] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2203.429321] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2208.598710] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2208.835397] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2213.816235] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2214.061351] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2215.906956] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2215.908200] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2215.917643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2215.924304] LustreError: Skipped 1 previous similar message [ 2215.951977] Lustre: Skipped 1 previous similar message [ 2230.240170] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2230.390577] Lustre: server umount lustre-OST0000 complete [ 2236.390361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2236.422140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2236.425146] Lustre: Skipped 5 previous similar messages [ 2241.102713] Lustre: server umount lustre-MDT0000 complete [ 2241.505557] LustreError: 76429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2245.300422] LustreError: 76411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788832489 with bad export cookie 2224209514962654973 [ 2245.314678] LustreError: 76411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2245.316262] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2245.644256] Lustre: server umount lustre-MDT0001 complete [ 2254.547925] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 21:54:56 (1788832496) [ 2261.182652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2261.667305] LustreError: 80801:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2261.693874] LustreError: 80801:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2261.761243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2266.152955] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2267.106474] LustreError: 80802:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2272.229751] LustreError: 80801:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2273.745227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2278.363632] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2285.866507] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2290.863764] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2297.040134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2297.409424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2297.414226] Lustre: Skipped 1 previous similar message [ 2303.574785] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2312.720920] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2317.472567] Lustre: 83471:0:(genops.c:1774:obd_export_evict_by_uuid()) MGS: evicting 0801fb3a-e1e7-463e-8505-9ef8a8b797a8 at adminstrative request [ 2321.896784] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2321.919887] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2332.141418] Lustre: MGS: Client 0801fb3a-e1e7-463e-8505-9ef8a8b797a8 (at 0@lo) reconnecting [ 2332.158870] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x1eddfac4fd649a6f to 0x1eddfac4fd649d0f [ 2332.186707] Lustre: MGC192.168.202.148@tcp: Connection restored to 0@lo (at 0@lo) [ 2355.999567] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2356.235667] Lustre: lustre-OST0001: new disk, initializing [ 2356.240583] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2356.248473] Lustre: 83866:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2363.211614] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2366.571162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2366.580041] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2366.732996] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2372.820566] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2377.193408] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2377.196564] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2377.202936] LustreError: Skipped 1 previous similar message [ 2377.203660] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2377.233671] Lustre: Skipped 2 previous similar messages [ 2381.845512] Lustre: server umount lustre-OST0001 complete [ 2382.315989] LustreError: 84352:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2382.335529] LustreError: 84352:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2385.379395] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2391.528196] Lustre: server umount lustre-OST0000 complete [ 2397.160019] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2397.168048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2397.186956] Lustre: Skipped 2 previous similar messages [ 2401.828902] Lustre: server umount lustre-MDT0000 complete [ 2402.787294] LustreError: 80802:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2402.802344] LustreError: 80802:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2405.262601] LustreError: 83865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788832648 with bad export cookie 2224209514962656527 [ 2405.272275] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2405.277779] LustreError: 83865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2405.645532] Lustre: server umount lustre-MDT0001 complete [ 2414.103548] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 21:57:35 (1788832655) [ 2415.842581] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2417.821732] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 21:57:39 (1788832659) [ 2419.873826] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2421.314874] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 21:57:43 (1788832663) [ 2422.835422] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2425.395088] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 21:57:46 (1788832666) [ 2427.257531] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2428.887380] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 21:57:50 (1788832670) [ 2430.362705] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2436.383259] loop: module loaded [ 2436.431104] loop0: detected capacity change from 0 to 4294967296 [ 2436.469035] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2438.435138] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2442.963711] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2448.050778] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2481.415531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2481.483420] LustreError: 87990:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2481.491297] LustreError: 87990:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2481.499778] LustreError: 87990:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2481.510569] LustreError: 87990:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2499.760192] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [ 2510.557458] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 2552.707533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2552.965445] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2552.996151] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2553.062799] Lustre: lustre-MDT0000: new disk, initializing [ 2553.117887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2553.126847] Lustre: Skipped 1 previous similar message [ 2553.142657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2558.697825] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2570.539502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2570.646742] Lustre: 90860:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2570.932777] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2575.550643] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2585.724684] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2591.715308] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2598.246067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2598.517949] Lustre: 92830:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2605.293562] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2605.617175] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2605.742188] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2616.434586] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2623.073773] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2623.499530] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2630.467720] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2630.744427] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2636.257822] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2636.263975] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2636.273152] Lustre: Skipped 1 previous similar message [ 2636.277884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2636.288870] Lustre: Skipped 6 previous similar messages [ 2639.111083] Lustre: server umount lustre-OST0000 complete [ 2650.085235] Lustre: server umount lustre-MDT0000 complete [ 2652.141733] LustreError: 90869:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2652.168884] LustreError: 90869:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2654.115327] LustreError: 90852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788832897 with bad export cookie 2224209514962657136 [ 2654.121379] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2654.125481] LustreError: 90852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2654.416807] Lustre: server umount lustre-MDT0001 complete [ 2662.503927] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 22:01:44 (1788832904) [ 2669.155676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2674.629575] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2684.537444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2685.162193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-60 [ 2685.172171] Lustre: Skipped 3 previous similar messages [ 2689.979907] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2697.987540] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2704.246097] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2712.112541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2720.228603] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2728.499362] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2734.601322] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2734.871080] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2739.733401] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2740.041690] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2747.259500] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2747.453670] Lustre: lustre-OST0001: new disk, initializing [ 2747.459775] Lustre: Skipped 2 previous similar messages [ 2747.466120] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2747.475368] Lustre: Skipped 3 previous similar messages [ 2747.484747] Lustre: 98629:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2755.527317] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2756.777264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2756.795761] Lustre: Skipped 2 previous similar messages [ 2756.803450] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2756.961057] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2765.781742] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2770.404847] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2770.652437] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2775.847250] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2776.186813] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2796.225636] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2822.470759] Lustre: Failing over lustre-MDT0000 [ 2823.020442] Lustre: server umount lustre-MDT0000 complete [ 2823.663172] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2823.669672] LustreError: 95986:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2823.693910] Lustre: Skipped 6 previous similar messages [ 2823.710713] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2823.736317] LustreError: 95986:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 2823.774405] LustreError: Skipped 1 previous similar message [ 2840.033885] Lustre: 66629:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788833067/real 1788833067] req@0000000029d10e10 x1875726424095616/t0(0) o400->MGC192.168.202.148@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788833083 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2840.069373] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2850.903098] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2850.907065] LDISKFS-fs (dm-0): recovery complete [ 2850.926361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2864.676895] LustreError: 66625:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007506d6f6 x1875726424107904/t0(0) o250->MGC192.168.202.148@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2865.282335] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2865.699413] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2865.701471] Lustre: 95254:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2870.244380] Lustre: 95258:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2870.259448] Lustre: 95258:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 1 previous similar message [ 2870.270208] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2870.298674] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2870.410386] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2870.754912] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2871.093234] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2871.109910] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 1 previous similar message [ 2871.432494] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2871.448926] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 1 previous similar message [ 2871.472897] Lustre: 101478:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2871.485873] Lustre: 101478:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 7 previous similar messages [ 2872.127840] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2872.140863] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 2 previous similar messages [ 2872.464096] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2872.479302] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 2 previous similar messages [ 2873.787307] Lustre: 101478:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2873.806239] Lustre: 101478:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 6 previous similar messages [ 2874.129065] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2874.148692] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 5 previous similar messages [ 2874.472331] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2874.478808] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 5 previous similar messages [ 2877.996927] Lustre: 101478:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2878.012895] Lustre: 101478:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 12 previous similar messages [ 2878.333243] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2878.346071] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 12 previous similar messages [ 2878.664408] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2878.674314] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 12 previous similar messages [ 2886.034742] Lustre: 101478:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2886.068253] Lustre: 101478:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 24 previous similar messages [ 2886.393261] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2886.416751] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 24 previous similar messages [ 2886.752232] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2886.759659] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 24 previous similar messages [ 2902.040628] Lustre: 101478:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2902.053587] Lustre: 101478:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 49 previous similar messages [ 2902.698749] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2902.720239] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 50 previous similar messages [ 2903.048173] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2903.059814] LustreError: 101478:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 50 previous similar messages [ 2925.184158] Lustre: 101478:0:(ldlm_lib.c:2419:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2925.195862] Lustre: 101478:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2925.202332] Lustre: 101478:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 2c84c44a-fb71-4379-964b-3dc86cb5f4ff@192.168.202.48@tcp [ 2925.211263] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2925.216092] Lustre: 101478:0:(ldlm_lib.c:1939:abort_req_replay_queue()) @@@ aborted: req@00000000d5e37aa3 x1875726408158208/t0(8589934768) o35->2c84c44a-fb71-4379-964b-3dc86cb5f4ff@192.168.202.48@tcp:354/0 lens 392/0 e 0 to 0 dl 1788833179 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2925.236373] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2925.262592] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2925.270348] Lustre: lustre-MDT0000: Denying connection for new client 2c84c44a-fb71-4379-964b-3dc86cb5f4ff (at 192.168.202.48@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 2925.313849] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2925.318375] Lustre: Skipped 2 previous similar messages [ 2925.344333] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2925.373461] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:45 to 0x2c0000401:545) [ 2925.375600] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:77 to 0x280000401:577) [ 2939.887333] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2939.907200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2939.921437] Lustre: Skipped 3 previous similar messages [ 2944.662390] Lustre: server umount lustre-OST0000 complete [ 2952.166392] LustreError: 97080:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2952.187763] LustreError: 97080:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 55 previous similar messages [ 2955.919445] Lustre: server umount lustre-OST0001 complete [ 2961.378602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2961.388916] Lustre: Skipped 4 previous similar messages [ 2974.176287] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2974.605885] Lustre: server umount lustre-MDT0000 complete [ 2980.479387] LustreError: 95240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833224 with bad export cookie 2224209514962708355 [ 2980.489044] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2980.493338] LustreError: 95240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2980.769344] Lustre: server umount lustre-MDT0001 complete [ 2993.096666] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 22:07:14 (1788833234) [ 3003.280625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3003.286177] Lustre: *** cfs_fail_loc=197, val=0*** [ 3018.788055] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3023.904520] LustreError: 103455:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.148@tcp: failed processing log, type 4: rc = -110 [ 3029.113770] Lustre: *** cfs_fail_loc=197, val=0*** [ 3029.118611] Lustre: Skipped 65 previous similar messages [ 3049.696479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3049.708788] Lustre: Skipped 3 previous similar messages [ 3057.665888] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3067.106633] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3070.316911] Lustre: server umount lustre-OST0000 complete [ 3082.219604] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 22:08:42 (1788833322) [ 3104.298928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3110.222219] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3119.189582] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3123.201639] LustreError: 105108:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3123.221842] LustreError: 105108:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3123.235584] LustreError: 105108:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 3123.486172] Lustre: server umount lustre-MDT0000 complete [ 3160.592528] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [ 3171.485729] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 3208.882227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3209.202777] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3209.241689] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3209.334634] Lustre: lustre-MDT0000: new disk, initializing [ 3209.435738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3214.088954] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3224.454716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3224.676350] Lustre: 109296:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3224.945685] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3229.766170] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3240.626950] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3247.262191] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3254.711873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3255.127741] Lustre: 111263:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3261.701911] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3262.165941] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3271.885983] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3277.128397] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3277.420557] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3282.035387] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3282.315448] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3287.521058] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3287.526612] LustreError: Skipped 4 previous similar messages [ 3287.530707] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3287.540717] Lustre: Skipped 2 previous similar messages [ 3287.542114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3287.561369] Lustre: Skipped 14 previous similar messages [ 3290.526622] Lustre: server umount lustre-OST0000 complete [ 3300.682100] Lustre: server umount lustre-MDT0000 complete [ 3302.885218] LustreError: 109300:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3302.914109] LustreError: 109300:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 3305.146053] LustreError: 109287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833548 with bad export cookie 2224209514962718932 [ 3305.159180] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3305.167473] LustreError: 109287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3305.664959] Lustre: server umount lustre-MDT0001 complete [ 3315.264264] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 22:12:36 (1788833556) [ 3326.790671] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 3329.630903] Key type lgssc unregistered [ 3330.034981] LNet: 113751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3330.047632] LNetError: 113751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3330.063575] LNet: Removed LNI 192.168.202.148@tcp [ 3331.559502] Key type .llcrypt unregistered [ 3331.561156] Key type ._llcrypt unregistered [ 3354.938704] Key type ._llcrypt registered [ 3354.948304] Key type .llcrypt registered [ 3355.107907] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [ 3372.369138] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 3374.934491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3375.007449] alg: No test for adler32 (adler32-zlib) [ 3376.181954] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 3376.450745] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 3378.184617] Key type lgssc registered [ 3379.670116] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3426.509790] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3426.576415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3427.894152] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3427.950592] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3428.018602] Lustre: lustre-MDT0000: new disk, initializing [ 3428.158949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3428.180625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3433.016488] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3444.564422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3444.811315] Lustre: lustre-OST0000: new disk, initializing [ 3444.816394] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3444.821598] Lustre: Skipped 1 previous similar message [ 3444.826204] Lustre: 117921:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3444.931408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3451.865781] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3454.005203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3454.013619] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3454.046147] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3463.273743] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3463.530731] Lustre: lustre-OST0001: new disk, initializing [ 3463.539962] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3463.565513] Lustre: 118783:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3463.679417] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3469.374812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3469.385704] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3469.420573] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3470.420681] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3479.528130] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3479.538647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3479.547474] Lustre: Skipped 1 previous similar message [ 3479.563183] Lustre: Skipped 1 previous similar message [ 3484.641779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3484.652462] Lustre: Skipped 1 previous similar message [ 3485.233762] Lustre: server umount lustre-MDT0000 complete [ 3491.548437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3496.418737] LustreError: 119394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833740 with bad export cookie 13561892771573890178 [ 3496.429224] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3496.430882] LustreError: 119394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3496.493865] Lustre: server umount lustre-OST0000 complete [ 3500.687810] Lustre: server umount lustre-OST0001 complete [ 3509.694108] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 22:15:51 (1788833751) [ 3532.889953] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 22:16:14 (1788833774) [ 3542.538375] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 3565.799359] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 22:16:47 (1788833807) [ 3607.383181] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [ 3619.127797] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 3656.351708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3656.584812] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3656.611499] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3656.696401] Lustre: lustre-MDT0000: new disk, initializing [ 3656.765090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3656.777442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3660.244722] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3672.088750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3672.222749] Lustre: 126537:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3672.318530] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3672.330288] Lustre: Skipped 1 previous similar message [ 3672.430252] Lustre: lustre-MDT0001: new disk, initializing [ 3672.533509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3672.589131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3672.602917] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3678.440736] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3690.012845] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3697.022880] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3703.997445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3704.290924] Lustre: lustre-OST0000: new disk, initializing [ 3704.294199] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3704.299453] Lustre: 128506:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3704.398499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3711.002513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3711.010326] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3711.138564] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3711.923956] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3723.078248] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3729.744330] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3730.093185] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3736.686971] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3736.917898] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3741.667488] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3741.676888] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3741.701075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3742.184250] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3746.179458] Lustre: server umount lustre-OST0000 complete [ 3752.419613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3752.422038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3752.445756] Lustre: Skipped 2 previous similar messages [ 3752.459958] Lustre: Skipped 3 previous similar messages [ 3757.222652] Lustre: server umount lustre-MDT0000 complete [ 3757.551542] LustreError: 126548:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3757.583022] LustreError: 126548:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3761.563519] LustreError: 127346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788834005 with bad export cookie 13561892771573891403 [ 3761.576650] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3761.588980] LustreError: 127346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3761.938358] Lustre: server umount lustre-MDT0001 complete [ 3771.414985] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 22:20:12 (1788834012) [ 3779.492964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3780.348605] LustreError: 130920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3780.505252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3785.699514] LustreError: 130921:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3786.089337] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3790.824971] LustreError: 130920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3795.601426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3795.952337] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3800.933493] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3809.478878] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3815.678460] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3824.433554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3824.840949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3824.851080] Lustre: Skipped 1 previous similar message [ 3831.282586] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3838.514117] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3846.986906] Lustre: *** cfs_fail_loc=159, val=0*** [ 3863.662122] Lustre: lustre-MDT0000: Client 178d38e0-d243-4e24-bc44-6caa183139ef (at 192.168.202.48@tcp) reconnecting [ 3865.191562] Lustre: *** cfs_fail_loc=159, val=0*** [ 3865.198767] Lustre: Skipped 5 previous similar messages [ 3881.553518] Lustre: lustre-MDT0000: Client 178d38e0-d243-4e24-bc44-6caa183139ef (at 192.168.202.48@tcp) reconnecting [ 3902.945497] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3902.946099] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3902.956901] LustreError: Skipped 1 previous similar message [ 3902.958287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3902.992167] Lustre: Skipped 1 previous similar message [ 3904.842352] Lustre: server umount lustre-OST0000 complete [ 3909.093575] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3909.097120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3915.139496] Lustre: server umount lustre-MDT0000 complete [ 3918.305941] LustreError: 133587:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3918.336758] LustreError: 133587:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3919.621536] LustreError: 132764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788834163 with bad export cookie 13561892771573892285 [ 3919.624442] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3919.636785] LustreError: 132764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3919.958510] Lustre: server umount lustre-MDT0001 complete [ 3932.167885] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 3935.190185] Key type lgssc unregistered [ 3935.497908] LNet: 134702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3935.508434] LNetError: 134702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3935.519320] LNet: Removed LNI 192.168.202.148@tcp [ 3936.578529] Key type .llcrypt unregistered [ 3936.581417] Key type ._llcrypt unregistered [ 3961.309554] Key type ._llcrypt registered [ 3961.312929] Key type .llcrypt registered [ 3961.530813] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 3963.013068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3963.426664] alg: No test for adler32 (adler32-zlib) [ 3964.534661] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 3964.816566] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 3966.480188] Key type lgssc registered [ 3967.857775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3975.757333] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3975.782657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3977.454355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3981.855209] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3987.938698] LustreError: 136126:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3987.974635] LustreError: 136126:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3988.967466] LustreError: 136125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3991.535426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3991.905809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3996.552535] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 4004.736866] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4011.305376] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4019.286217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4019.622572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4023.737877] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4026.693557] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 4034.546886] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4047.624299] Lustre: *** cfs_fail_loc=159, val=0*** [ 4063.836912] Lustre: lustre-MDT0000: Client 5570d79f-0450-4b2e-97e8-696e9e32803e (at 192.168.202.48@tcp) reconnecting [ 4066.263676] Lustre: *** cfs_fail_loc=159, val=0*** [ 4082.816729] Lustre: lustre-MDT0001: Client 5570d79f-0450-4b2e-97e8-696e9e32803e (at 192.168.202.48@tcp) reconnecting [ 4085.140608] Lustre: *** cfs_fail_loc=159, val=0*** [ 4085.142532] Lustre: Skipped 3 previous similar messages [ 4101.760026] Lustre: lustre-MDT0001: Client 5570d79f-0450-4b2e-97e8-696e9e32803e (at 192.168.202.48@tcp) reconnecting [ 4117.336986] Lustre: 139182:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4120.666513] Lustre: *** cfs_fail_loc=159, val=0*** [ 4120.670721] Lustre: Skipped 4 previous similar messages [ 4136.069937] Lustre: lustre-MDT0000: Client 14699a6e-bae8-482c-9b37-ee3c01238c50 (at 192.168.202.48@tcp) reconnecting [ 4138.034259] Lustre: *** cfs_fail_loc=159, val=0*** [ 4138.045341] Lustre: Skipped 15 previous similar messages [ 4154.452855] Lustre: lustre-MDT0000: Client 14699a6e-bae8-482c-9b37-ee3c01238c50 (at 192.168.202.48@tcp) reconnecting [ 4156.384943] Lustre: 139476:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4166.120155] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4166.140454] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4166.147673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4166.160397] Lustre: Skipped 1 previous similar message [ 4166.184249] Lustre: Skipped 1 previous similar message [ 4170.623243] Lustre: server umount lustre-OST0000 complete [ 4174.871767] Lustre: server umount lustre-MDT0000 complete [ 4176.358331] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4176.361182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4176.378550] LustreError: 139242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4176.398485] Lustre: Skipped 1 previous similar message [ 4176.426099] LustreError: 139242:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4178.991027] LustreError: 136105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788834422 with bad export cookie 4051525030521255291 [ 4179.004700] LustreError: 136105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4179.006130] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4179.405118] Lustre: server umount lustre-MDT0001 complete [ 4192.139418] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 4195.103472] Key type lgssc unregistered [ 4195.388741] LNet: 140529:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4195.413234] LNetError: 140529:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4195.449352] LNet: Removed LNI 192.168.202.148@tcp [ 4196.546326] Key type .llcrypt unregistered [ 4196.547976] Key type ._llcrypt unregistered [ 4228.417720] Key type ._llcrypt registered [ 4228.422941] Key type .llcrypt registered [ 4228.565453] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 4229.753193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4229.811177] alg: No test for adler32 (adler32-zlib) [ 4230.914206] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 4231.167632] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 4232.904189] Key type lgssc registered [ 4234.462451] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4242.730803] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4242.752912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4244.256379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4248.820393] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 4253.669147] LustreError: 141943:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4253.696221] LustreError: 141943:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4258.790489] LustreError: 141942:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4258.940907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4259.388955] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4264.234460] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 4274.063254] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4279.555265] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4288.290766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4288.911361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4289.906833] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4296.367272] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 4304.525810] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4305.272103] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4311.451353] Lustre: 144527:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4315.263713] Lustre: 144576:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4316.811517] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4317.947876] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4322.504646] Lustre: 144722:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4323.663756] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4324.643881] Lustre: 144821:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4330.977916] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4330.998812] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4331.002888] LustreError: Skipped 1 previous similar message [ 4331.004567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4331.054553] Lustre: Skipped 1 previous similar message [ 4336.112930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4336.123623] Lustre: Skipped 2 previous similar messages [ 4336.455901] Lustre: server umount lustre-OST0000 complete [ 4341.218388] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4341.227028] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4341.237602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4341.280365] Lustre: Skipped 1 previous similar message [ 4346.338344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4346.355828] Lustre: Skipped 2 previous similar messages [ 4346.821535] Lustre: server umount lustre-MDT0000 complete [ 4350.815234] LustreError: 141922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788834594 with bad export cookie 2716485557527368960 [ 4350.820202] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4350.828279] LustreError: 141922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4351.187670] Lustre: server umount lustre-MDT0001 complete [ 4363.792544] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 4366.715126] Key type lgssc unregistered [ 4366.991289] LNet: 145884:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4367.018486] LNetError: 145884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4367.037401] LNet: Removed LNI 192.168.202.148@tcp [ 4368.008207] Key type .llcrypt unregistered [ 4368.010537] Key type ._llcrypt unregistered [ 4394.309451] Key type ._llcrypt registered [ 4394.314813] Key type .llcrypt registered [ 4394.510231] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 4395.869167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4395.948265] alg: No test for adler32 (adler32-zlib) [ 4397.062546] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 4397.315283] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 4399.000196] Key type lgssc registered [ 4400.312628] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4408.798894] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4408.826995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4410.395637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4415.656049] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 4420.584115] LustreError: 147297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4420.602139] LustreError: 147297:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4425.698745] LustreError: 147296:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4425.954723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4426.378337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4432.131355] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 4440.922658] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4448.039398] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4456.372989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4457.011039] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4464.230457] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 4471.357178] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4471.945036] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4480.562995] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4488.655898] Lustre: *** cfs_fail_loc=159, val=0*** [ 4504.196482] Lustre: lustre-MDT0000: Client 2497e8df-0bae-4d7d-9d48-661b49b145d9 (at 192.168.202.48@tcp) reconnecting [ 4522.466699] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4522.473097] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4522.492511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4526.562048] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4526.565268] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4526.597754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4526.611666] Lustre: Skipped 1 previous similar message [ 4528.134344] Lustre: server umount lustre-OST0000 complete [ 4533.730835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4533.744490] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4533.770663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4536.806397] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4536.807430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4536.838410] Lustre: Skipped 1 previous similar message [ 4539.107611] Lustre: server umount lustre-MDT0000 complete [ 4541.927885] LustreError: 149928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4541.971825] LustreError: 149928:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4543.430143] LustreError: 147277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788834787 with bad export cookie 9603576555828237444 [ 4543.453445] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4543.459618] LustreError: 147277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4543.949523] Lustre: server umount lustre-MDT0001 complete [ 4556.481740] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 4559.880595] Key type lgssc unregistered [ 4560.472824] LNet: 150989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4560.485965] LNetError: 150989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4561.518822] LNet: Removed LNI 192.168.202.148@tcp [ 4562.672140] Key type .llcrypt unregistered [ 4562.678213] Key type ._llcrypt unregistered [ 4590.717808] Key type ._llcrypt registered [ 4590.720183] Key type .llcrypt registered [ 4590.803520] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 4592.337572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4592.394185] alg: No test for adler32 (adler32-zlib) [ 4593.607526] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 4593.953398] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 4595.640516] Key type lgssc registered [ 4596.765602] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4607.083329] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4607.118774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4608.588403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4613.551219] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 4618.729740] LustreError: 152396:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4618.754779] LustreError: 152396:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4623.232279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4623.687797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4629.869259] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 4639.021415] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4646.589403] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4654.918746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4655.360399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4662.267991] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 4668.662805] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4669.335681] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4678.901047] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4698.236120] Lustre: lustre-MDT0001: evicting e377bc53-7ed9-40a5-b622-43991f4ae988 (at 192.168.202.48@tcp) by administrative request [ 4698.245837] Lustre: Skipped 1 previous similar message [ 4730.338809] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4730.343055] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4730.352201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4730.352207] Lustre: Skipped 1 previous similar message [ 4732.615904] Lustre: server umount lustre-OST0000 complete [ 4736.483493] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4736.494918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4736.511719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4736.525271] Lustre: Skipped 1 previous similar message [ 4740.582037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4740.585282] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4745.697621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4745.706577] Lustre: Skipped 2 previous similar messages [ 4750.304254] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4750.557302] Lustre: server umount lustre-MDT0000 complete [ 4750.825152] LustreError: 152396:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4750.842151] LustreError: 152396:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4754.296507] LustreError: 152377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788834998 with bad export cookie 287843470959863277 [ 4754.305157] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4754.314273] LustreError: 152377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4754.601598] Lustre: server umount lustre-MDT0001 complete [ 4766.775843] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 4769.365421] Key type lgssc unregistered [ 4769.748500] LNet: 156487:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4769.759736] LNetError: 156487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4769.781032] LNet: Removed LNI 192.168.202.148@tcp [ 4770.688254] Key type .llcrypt unregistered [ 4770.690037] Key type ._llcrypt unregistered [ 4842.514612] Key type ._llcrypt registered [ 4842.522374] Key type .llcrypt registered [ 4842.635839] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 4843.963516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4844.277709] alg: No test for adler32 (adler32-zlib) [ 4845.461552] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 4845.878852] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 4847.633240] Key type lgssc registered [ 4848.892690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4857.876565] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4857.901602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4859.505888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4864.285953] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 4870.121213] LustreError: 158974:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4870.138602] LustreError: 158974:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4874.688297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4875.340293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4881.017384] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 4889.748502] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4897.805573] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4906.643374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4907.038842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4914.799986] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 4919.372624] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 4923.906077] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4929.593135] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 4939.760704] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4939.762778] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4939.767506] LustreError: Skipped 1 previous similar message [ 4939.768293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4939.801282] Lustre: Skipped 1 previous similar message [ 4943.333493] Lustre: server umount lustre-OST0000 complete [ 4946.915211] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4946.933136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4946.952311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4946.964809] Lustre: Skipped 1 previous similar message [ 4949.994285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4949.995487] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4955.108586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4955.129299] Lustre: Skipped 2 previous similar messages [ 4960.226906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4960.241755] Lustre: Skipped 1 previous similar message [ 4961.248803] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4961.448825] Lustre: server umount lustre-MDT0000 complete [ 4965.056733] LustreError: 158953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788835208 with bad export cookie 3785039253287164894 [ 4965.068067] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4965.074249] LustreError: 158953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4965.437522] Lustre: server umount lustre-MDT0001 complete [ 4976.165169] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 4979.000659] Key type lgssc unregistered [ 4979.270069] LNet: 162538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4979.284060] LNetError: 162538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4979.312122] LNet: Removed LNI 192.168.202.148@tcp [ 4980.360845] Key type .llcrypt unregistered [ 4980.364225] Key type ._llcrypt unregistered [ 5007.828773] Key type ._llcrypt registered [ 5007.830467] Key type .llcrypt registered [ 5007.955550] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 5009.117536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5009.305509] alg: No test for adler32 (adler32-zlib) [ 5010.383587] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 5010.647509] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 5012.344190] Key type lgssc registered [ 5013.772651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5033.505778] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 22:41:14 (1788835274) [ 5053.921678] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [ 5065.706547] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 5104.884744] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5104.928828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5106.300789] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5106.350618] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5106.470936] Lustre: lustre-MDT0000: new disk, initializing [ 5106.600448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5106.630946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5111.443724] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 5122.775480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5122.891484] Lustre: 166799:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5122.953709] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5122.958320] Lustre: Skipped 1 previous similar message [ 5123.020367] Lustre: lustre-MDT0001: new disk, initializing [ 5123.122384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5123.176600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5123.197426] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5128.606290] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 5139.422420] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5145.851307] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5153.865032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5154.268504] Lustre: lustre-OST0000: new disk, initializing [ 5154.276823] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5154.287593] Lustre: 168768:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5154.402141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5161.421436] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 5162.148911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5162.154439] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5162.344296] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5172.309501] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5182.951402] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5182.955365] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5182.981837] LustreError: Skipped 1 previous similar message [ 5182.983278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5183.004271] Lustre: Skipped 1 previous similar message [ 5188.173234] Lustre: server umount lustre-OST0000 complete [ 5193.187306] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5193.193690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5193.213893] Lustre: Skipped 1 previous similar message [ 5198.314625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5198.324921] Lustre: Skipped 1 previous similar message [ 5199.222654] Lustre: server umount lustre-MDT0000 complete [ 5203.510841] LustreError: 168766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788835447 with bad export cookie 8833700813649966919 [ 5203.512447] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5203.520142] LustreError: 168766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5203.868403] Lustre: server umount lustre-MDT0001 complete [ 5215.888821] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 5218.581542] Key type lgssc unregistered [ 5219.008897] LNet: 170670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5219.034173] LNetError: 170670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5220.088920] LNet: Removed LNI 192.168.202.148@tcp [ 5221.235442] Key type .llcrypt unregistered [ 5221.240331] Key type ._llcrypt unregistered [ 5247.932059] Key type ._llcrypt registered [ 5247.933727] Key type .llcrypt registered [ 5248.028097] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 5249.306179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5249.337944] alg: No test for adler32 (adler32-zlib) [ 5250.435624] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 5250.701168] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 5252.400225] Key type lgssc registered [ 5253.555855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5262.564882] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5262.624891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5264.412960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5272.187236] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 5274.597589] LustreError: 172084:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5274.618036] LustreError: 172084:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5279.718985] LustreError: 172085:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5284.423264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5284.860114] LustreError: 172085:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5285.658792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5291.660381] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 5300.933960] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5309.117672] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5317.871782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5318.451245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5324.947407] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 5333.316415] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5334.841097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5340.178523] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5340.194452] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5340.204810] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5340.212739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5340.234705] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5341.516598] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5341.541626] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5341.550163] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5341.560722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5341.585793] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5342.884404] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5342.915867] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5342.928545] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5342.938791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5342.957325] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5343.956832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5345.035993] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5345.042175] Lustre: Skipped 1 previous similar message [ 5345.060562] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5345.074278] Lustre: Skipped 1 previous similar message [ 5345.077668] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5345.084266] Lustre: Skipped 1 previous similar message [ 5345.091645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5345.100449] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5345.104238] Lustre: Skipped 1 previous similar message [ 5345.994843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5347.101559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5348.252531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5349.383052] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5349.389958] Lustre: Skipped 3 previous similar messages [ 5349.418320] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5349.432923] Lustre: Skipped 3 previous similar messages [ 5349.443350] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5349.454166] Lustre: Skipped 3 previous similar messages [ 5349.462362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5349.501631] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5349.505306] Lustre: Skipped 3 previous similar messages [ 5350.686860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5351.700679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5352.783794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5353.758432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5354.770841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5355.537774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5356.413270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5357.401703] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5357.410869] Lustre: Skipped 7 previous similar messages [ 5357.434468] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5357.450604] Lustre: Skipped 7 previous similar messages [ 5357.458856] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5357.461847] Lustre: Skipped 7 previous similar messages [ 5357.472362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5358.559304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5358.575444] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5358.578635] Lustre: Skipped 8 previous similar messages [ 5359.639154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5360.432208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5361.513190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5362.633893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5363.717679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5364.720253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5365.720189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5366.594262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5367.500810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5368.347209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5369.201373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5370.092166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5371.024871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5371.892240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5372.728162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5373.791403] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5373.807091] Lustre: Skipped 16 previous similar messages [ 5373.855477] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5373.883267] Lustre: Skipped 16 previous similar messages [ 5373.896165] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5373.902554] Lustre: Skipped 16 previous similar messages [ 5373.906137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5374.996484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5375.018636] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5375.029706] Lustre: Skipped 16 previous similar messages [ 5376.135846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5377.503126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5378.745754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5379.666946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5380.995164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5382.121402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5383.313943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5384.621601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5385.874513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5386.936840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5388.278982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5389.465899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5390.512403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5391.797981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5392.945839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5393.922217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5395.261805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5396.646262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.976050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5399.535791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5400.774620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5402.173062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5403.173280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5404.591591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5405.789247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5406.955252] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5406.960972] Lustre: Skipped 26 previous similar messages [ 5406.990332] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5406.998637] Lustre: Skipped 26 previous similar messages [ 5407.001763] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5407.006722] Lustre: Skipped 26 previous similar messages [ 5407.010229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5407.019577] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5407.028212] Lustre: Skipped 25 previous similar messages [ 5407.922396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5409.469653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5411.399704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5412.837233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5414.210182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5415.405956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5416.795072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5417.803447] LustreError: 172104:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5417.812453] LustreError: 172104:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5417.828364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.832063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5419.698059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5420.371171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5421.162447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5421.989513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5422.596948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5423.382143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5424.193880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5425.134040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5426.230243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5427.188457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5428.460474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5429.782978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5431.511111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5432.717264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5433.974124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5435.071977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5436.307410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5437.450074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5438.316855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5439.345417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5440.562177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5441.498042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5442.541164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5443.513595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5444.592060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5445.451345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5446.683850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5447.585991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5448.704968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5449.792428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5450.927223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5452.036303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5453.204348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5454.513718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5455.656149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5456.472562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5457.606505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5458.501137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5459.618512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5460.469547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.619414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5462.514731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5463.508187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5464.430949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5465.461281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5466.351884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.240718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5468.299179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.278143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5470.335198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5471.420277] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5471.436418] Lustre: Skipped 59 previous similar messages [ 5471.455111] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5471.465338] Lustre: Skipped 59 previous similar messages [ 5471.472855] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5471.475876] Lustre: Skipped 59 previous similar messages [ 5471.480332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5471.487646] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5471.490440] Lustre: Skipped 59 previous similar messages [ 5472.617516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.708297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5474.972780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5476.035301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5477.117041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5478.339594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5479.667227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5480.673453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.463825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.592192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5483.709442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.733661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.751041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5486.928919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5487.916639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.954968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.953428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.146254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.399710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.405400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5494.549428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.339464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5496.263560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.052232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.974469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5498.835849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.919375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.970339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.991392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5503.215897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.071865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.496665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.758994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.143909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5509.169836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.434504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5511.560527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.481766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.603139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5514.601094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.495899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5516.538884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.546350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.379810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5519.342661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.218786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.154600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5522.008547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.132696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.028062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.887477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5525.638365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5526.629851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.650924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.643370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.540811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5530.478970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.584695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.415634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5533.373943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.363810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5535.426331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.312289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5537.306776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.207532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5539.480239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.642796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5541.561696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.463214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.368664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.314552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.381295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.469775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.508818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.631512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.509986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.557255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.650652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.644891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.589343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.610613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.633515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5556.488245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.381918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.361705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.400445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.495058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.656274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.413396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.370381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.273875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5565.199233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.019187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.903430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.698944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.668305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.753311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.844191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.249289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.666381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.895624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.824243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.026902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.073499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.327430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.461613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.323952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.742809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.311581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.976446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.165914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.410284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.320611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.229268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.081837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.442515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.591096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.599548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.588355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.661259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.741776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.876385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.576665] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5599.579521] Lustre: Skipped 122 previous similar messages [ 5599.592663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5599.599923] Lustre: Skipped 122 previous similar messages [ 5599.603195] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5599.606676] Lustre: Skipped 122 previous similar messages [ 5599.609049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.615945] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5599.620389] Lustre: Skipped 122 previous similar messages [ 5600.303160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.125852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.099684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.389117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.404745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.561781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.662439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.701324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.895243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.696719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.751065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.864339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.025760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.209379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.269207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.282549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.282481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.183432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.893660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.702920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.946648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.847653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.923526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.670608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.747560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.494771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.409627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.507913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.347931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.447808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.536976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.549382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.766843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.705626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.871804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.983658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.066069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.127613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.058281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.046234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.138341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.200329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.320550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.435785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.540812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.407479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.454936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.482282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.523676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.607266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.038250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.423617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.495377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.479287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.538917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.464334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.567062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.578792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.790492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.560079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.632395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.496189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.364854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.231236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.308308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.381143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.464390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.313083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.278985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.269431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.237362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.236589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.361876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.590247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.007775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.007486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.928512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.973695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.078809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.383503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.514884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.720100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.579856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.780876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.824043] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.746962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.811322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.023667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.930557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.894760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.816613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.865058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.891626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.820214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.736694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.703816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.701687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.683770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.807449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.085385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.141904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.327252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.443412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.367580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.361117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.193483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.307567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.266585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5712.073474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.135749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.100801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.965979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.816700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.755707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.899871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.934745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.347752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.405729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.601900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.510963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.463575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.722666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.757439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.775428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.955926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.156524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.455673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.552034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.898634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.061322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.178337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.346682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.421355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.469844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5740.619209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.703965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.657924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.651728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.676237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.596716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.501813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.388162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.367233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.359127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5750.198345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.187528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.186184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.172894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.850490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.659341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.608304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.604853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.629930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.686630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.516504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.367181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.281122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.260707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.140455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.999508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.738917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.647859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.584528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.531672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.434641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.409130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.287409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.133033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.962494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.072169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.947370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.980123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.989363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.949157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.252308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.448706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.697512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.113712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.396505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.548162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.251320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.411936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.501531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.493265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.770130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.758216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.145363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.351765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.574433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.819257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.193062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.589158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.614496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.690908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.886794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.878594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.923751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.999913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.942651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.858383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.775915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.761690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.796304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.745112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.625293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.436368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.277409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5814.378342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.463728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.469075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.395943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.207632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.526809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.553655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.978832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.015583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.040808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.431851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.327723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.713220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.002385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.463925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.591812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.873379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.182940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5835.298480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.015333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.327535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.033630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.169526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.444016] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.431613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.439699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5844.325237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.203943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.100229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.245888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.178695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.431363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.405733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.645407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.524069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.661233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.503225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.527363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.341105] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5856.347570] Lustre: Skipped 245 previous similar messages [ 5856.369139] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5856.378440] Lustre: Skipped 245 previous similar messages [ 5856.382502] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5856.386823] Lustre: Skipped 245 previous similar messages [ 5856.390587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.399592] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5856.405847] Lustre: Skipped 245 previous similar messages [ 5857.187618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.226232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.103338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.940209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.049451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.269165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.814971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.878689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.877377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.144995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.290525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.248963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.190556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.210901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.205276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.156325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.054837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5875.082883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.082324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.266281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.372438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.341540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.456724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5881.393557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5882.456885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.351450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.309461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.994149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.835519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.855348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.621941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.533910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.285517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.176921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.089666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.250414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.099451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.071271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.039070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.944432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.725699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.585222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.407633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.265466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.940568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.768227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5901.576483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.456920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.234571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.003140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.991379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.979829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.831250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.778041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.547145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.438035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.338861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.183371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.059292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.938840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.757204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5914.939980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.091130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.394929] LustreError: 172104:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5917.397992] LustreError: 172104:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5917.407768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.467120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5919.647221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5921.093800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.165687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5923.184609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.055163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.241464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.245062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5927.095188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5927.993939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5928.843730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5929.626334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5930.455046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.353775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.415853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5933.388658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5934.443746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5935.696613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5936.882205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5938.044584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.171560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.424479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5941.366723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.392508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5943.345268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5944.331866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5945.513647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.700478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5947.539241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.671801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.803350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5950.803646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5951.682971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.041688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5954.303905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5955.603465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5956.548256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5957.531397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5958.366883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5959.247155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5959.920933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5960.897496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5961.853728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5963.017749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5964.022840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5965.086529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5966.233984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.089860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5968.029171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5968.823177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5969.777629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5970.830115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5971.780752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5972.769350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5973.664171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5974.580204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.553284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.711269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5977.640669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.558990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.405954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5980.384033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5981.192717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5982.405124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5983.472466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.480204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.011981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.949702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.216908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5989.621110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5990.498088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.643761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5992.726734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5993.711890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5994.792579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5995.789426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5997.125313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.256094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5999.597235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.875726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6001.859509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6002.878530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6004.219198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6005.386905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.305110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6007.295411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6008.499325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6009.661880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6010.855735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6012.129298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6013.258538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.485871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6015.474418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6016.742321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6018.124567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6019.708805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6020.968525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6022.440522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6023.676546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6025.001234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6025.980109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6027.424477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6028.109468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6028.789688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6029.726803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6030.652081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6031.543364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6032.713392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.001634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6035.268279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6036.355175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6037.507034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.306214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6039.018991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6039.620106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6040.552167] LustreError: 172104:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6040.558658] LustreError: 172104:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6040.570045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6041.316480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6042.258963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6043.448080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6044.400246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6045.639664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6046.407443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6047.218277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6047.922586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6048.845462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6049.637427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6050.454579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6051.522804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6052.522900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6053.597901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6054.728409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6055.693813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.991715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6058.486404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6059.692097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6060.908065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6062.023200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6062.899264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6063.885161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.878150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6065.959764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6067.089158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6067.985926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6068.888170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6069.721972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6070.846521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6071.886051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6072.897217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6073.855230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6074.715200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.626881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6076.678907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6077.738412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6078.818754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6079.862616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6080.830694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6081.826321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6082.639692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6083.605053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6084.372463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6085.418359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6086.161995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6087.168780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6088.035507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6089.134741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6089.995393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6090.883840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6091.922639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6092.787170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6093.682176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6094.621515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6095.317913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6096.208607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6096.953452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6097.909587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6098.763641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6099.684670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6100.674872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6101.685551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6102.605485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6103.501964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6104.594780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6105.908080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6106.709705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6107.596788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6108.520121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6109.836220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6110.725950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6111.699648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6112.665169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6113.724483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6114.591966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6115.657688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6116.333871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6117.243127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6118.141781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6119.120856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6120.182724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6121.202667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6122.131397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6122.962954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6123.894389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6124.777475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6125.821070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6126.701383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6127.518413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6128.348993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6129.202643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6130.120811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6130.977556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6131.911698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6132.658814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6133.730898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6134.759993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6135.580788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6136.549280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6137.820110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6138.711152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6139.679405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6140.399179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6141.218985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6142.230549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6142.940540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6143.698228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6144.814797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6145.733906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6146.869628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6148.143557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6149.367816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6150.133639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6151.296078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6152.480896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6153.904129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6155.052562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6156.835726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6158.400825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6159.748084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6161.264183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6162.641921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6164.064337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6165.017217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6166.037415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6166.939669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6168.056052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6168.927375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6170.006804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6170.984047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6172.043433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6173.241191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6174.282590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6175.546179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6176.593577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6177.512297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6178.554342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6179.587905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6180.505936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6181.606527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6182.620323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6183.849240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6185.029938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6186.334705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6187.666673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6188.777427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6189.956304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6190.975432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6191.950753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6193.055783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6194.055455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6195.254500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6196.139437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6197.394943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6198.521719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6199.250880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6200.699591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6201.974479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6203.162779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6204.415284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6205.701769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6206.837742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6207.925857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6208.932260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6210.145654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6211.682354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6213.133172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6214.113791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6215.436815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6216.581705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6217.890221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6219.191735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6220.305908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6221.501295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6222.493789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6223.628656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6224.565100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6225.939354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6226.690294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6227.850532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6228.835612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6229.849191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6230.992045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6232.332067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6233.405194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6234.594863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6235.711773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6237.033434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6238.274916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6239.461606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6240.435283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6241.446355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6242.430592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6243.491867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6244.555443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6245.388671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6246.514427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6247.450437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6248.473241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6249.492782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6250.466062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6251.503211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6252.532854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6253.834911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6254.914026] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6256.092180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6256.996870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6257.960874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6259.088321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6260.151903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6261.129465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6262.310087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6263.451682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6264.305614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6265.403295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6266.363778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6267.531659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6268.536560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6269.597413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6270.759271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6272.078395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6272.973599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6273.942632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6274.828550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6276.060300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6277.142625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6278.152768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6279.224919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6280.240882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6281.330967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6282.132949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6283.074759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6284.275042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6285.609959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6286.758659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6287.874939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6288.964722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6290.021292] LustreError: 172104:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6290.030316] LustreError: 172104:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6290.038354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6291.093396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6292.295960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6293.318743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6294.560539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6295.589953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6296.755901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6298.086455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6299.247063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6300.418136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6301.489567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6302.574609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6303.563683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6304.533977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6305.595464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6306.643287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6307.895658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6309.184652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6310.347827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6311.809116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6313.202574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6314.314135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6315.238359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6316.279093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6317.266133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6318.323017] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6319.458860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6320.444147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6321.450696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6322.363548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6323.602418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6324.693277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6325.840507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6326.895546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6327.949364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6329.180788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6330.057921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6331.169589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6332.432403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6333.509499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6334.565873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6335.559715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6336.478918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6337.510952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6338.771899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6340.281464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6341.621312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6343.058374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6344.056166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6345.527446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6346.493025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6347.675353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6348.517263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6349.460726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6350.391098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6351.346081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6352.256196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6353.510145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6354.507121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6355.738377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6356.977137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6358.258610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6359.843719] LustreError: 172104:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6359.861831] LustreError: 172104:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6359.874538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6360.873862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6362.069692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6363.233427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6364.182846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6365.582144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6366.763396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6367.983725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6369.087095] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6369.094641] Lustre: Skipped 488 previous similar messages [ 6369.112876] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6369.124971] Lustre: Skipped 488 previous similar messages [ 6369.137127] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6369.143560] Lustre: Skipped 488 previous similar messages [ 6369.148385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6369.157056] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6369.159561] Lustre: Skipped 488 previous similar messages [ 6370.153827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6371.280093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6372.460531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6373.511951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6374.465076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6375.680203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6376.960971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6378.064658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6379.096395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6380.166402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6381.165932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6382.330687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6383.301807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6384.377096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6385.370742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6386.241868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6387.533633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6388.708567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6389.757744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6390.675765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6391.676408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6772.852191] LustreError: 173907:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6772.872180] LustreError: 172104:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 7161.832474] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7161.843418] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7161.873407] Lustre: Skipped 21 previous similar messages [ 7161.910253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7163.876049] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7163.893741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7164.909685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7164.927557] Lustre: Skipped 1 previous similar message [ 7167.875499] Lustre: server umount lustre-OST0000 complete [ 7174.627800] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7174.649942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7178.243516] Lustre: server umount lustre-MDT0000 complete [ 7180.261757] LustreError: 172085:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7180.296034] LustreError: 172085:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7182.720324] LustreError: 172066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837426 with bad export cookie 11299285432153989188 [ 7182.724037] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7182.740942] LustreError: 172066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7183.175394] Lustre: server umount lustre-MDT0001 complete [ 7195.396408] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 7197.735794] Key type lgssc unregistered [ 7198.033597] LNet: 227324:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7198.046844] LNetError: 227324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7198.059187] LNet: Removed LNI 192.168.202.148@tcp [ 7198.852973] Key type .llcrypt unregistered [ 7198.855397] Key type ._llcrypt unregistered [ 7222.379901] Key type ._llcrypt registered [ 7222.381781] Key type .llcrypt registered [ 7222.478875] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 7223.318057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7223.404872] alg: No test for adler32 (adler32-zlib) [ 7224.471343] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 7224.680616] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 7226.352172] Key type lgssc registered [ 7227.282614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7235.601929] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7235.620761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7237.131181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7241.498973] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7246.306640] LustreError: 228739:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7246.324857] LustreError: 228739:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7249.682614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7250.080212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7254.338713] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7261.590282] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7266.706986] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7273.547624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7273.811381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7280.268148] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7281.969038] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 7288.085055] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7294.461512] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7296.606906] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7296.616083] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 7296.621382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7296.635034] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7296.637775] LustreError: 228759:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 7300.577906] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7300.586906] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7300.608356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7301.188930] Lustre: lustre-OST0000: Not available for connect from 192.168.202.48@tcp (stopping) [ 7302.627764] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7302.627930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7302.656573] Lustre: Skipped 1 previous similar message [ 7306.290186] Lustre: lustre-OST0000: Not available for connect from 192.168.202.48@tcp (stopping) [ 7311.394173] Lustre: lustre-OST0000: Not available for connect from 192.168.202.48@tcp (stopping) [ 7311.405181] Lustre: Skipped 2 previous similar messages [ 7313.377038] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7313.609248] Lustre: server umount lustre-OST0000 complete [ 7316.672134] LustreError: 228759:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 7349.823256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7350.252361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7357.855569] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7365.863366] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7367.703714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7367.721424] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7367.722325] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7367.739374] Lustre: Skipped 1 previous similar message [ 7368.862748] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 7375.317709] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9199cafa3000.ost_server_uuid 50 [ 7376.877299] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9199cafa3000.ost_server_uuid in FULL state after 0 sec [ 7383.013916] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7383.022695] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7383.035357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7383.052964] Lustre: Skipped 1 previous similar message [ 7383.063668] Lustre: Skipped 2 previous similar messages [ 7387.083494] Lustre: server umount lustre-OST0000 complete [ 7393.258183] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7393.258912] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7393.276276] Lustre: Skipped 1 previous similar message [ 7397.331382] Lustre: server umount lustre-MDT0000 complete [ 7398.369606] LustreError: 228736:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7398.402318] LustreError: 228736:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7400.592072] LustreError: 228720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837644 with bad export cookie 3923818572757872751 [ 7400.594862] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7400.629206] LustreError: 228720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7401.165254] Lustre: server umount lustre-MDT0001 complete [ 7411.267852] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 7413.738486] Key type lgssc unregistered [ 7414.039838] LNet: 233946:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7414.049675] LNetError: 233946:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7415.079151] LNet: Removed LNI 192.168.202.148@tcp [ 7415.861115] Key type .llcrypt unregistered [ 7415.863929] Key type ._llcrypt unregistered [ 7444.748189] Key type ._llcrypt registered [ 7444.750192] Key type .llcrypt registered [ 7444.864028] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 7446.032352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7446.053594] alg: No test for adler32 (adler32-zlib) [ 7447.107970] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 7447.344979] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 7448.992179] Key type lgssc registered [ 7450.279720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7459.316887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7459.325729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7460.735324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7465.178907] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7471.082179] LustreError: 235670:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7471.101583] LustreError: 235670:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7473.441734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7473.895256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7478.412285] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7481.489515] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7489.281056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7489.668146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7496.956561] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7506.120029] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 7507.850265] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7508.086899] Lustre: lustre-OST0001: new disk, initializing [ 7508.093628] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7508.102324] Lustre: 237978:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7508.205393] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7514.320363] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7516.376896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7516.392904] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7516.555540] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7524.165675] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7533.491208] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7545.464468] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 7545.475273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7545.492159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7545.912344] Lustre: server umount lustre-MDT0000 complete [ 7547.364647] LustreError: 235671:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7547.367425] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7547.402518] Lustre: Skipped 2 previous similar messages [ 7548.931680] LustreError: 238866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837792 with bad export cookie 7407167350910776489 [ 7548.937079] LustreError: 236408:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200001b72:0x1:0x0]: rc = -5 [ 7548.942731] LustreError: 238866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7548.947256] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7549.200260] Lustre: server umount lustre-MDT0001 complete [ 7563.025358] Lustre: server umount lustre-OST0000 complete [ 7577.244352] Lustre: server umount lustre-OST0001 complete [ 7602.248875] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 7615.261353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7615.719700] LustreError: 241082:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7615.741701] LustreError: 241082:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7615.835325] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7619.906360] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7621.096891] LustreError: 241083:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7626.215157] LustreError: 241082:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7627.917614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7628.379143] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7633.970953] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7636.879768] Lustre: 242224:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7642.738368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7649.385120] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7652.395086] LustreError: 242579:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7652.406796] Lustre: mylustre-OST0000: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 7657.074775] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7657.393225] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7657.401600] Lustre: Skipped 1 previous similar message [ 7660.490773] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49760) [ 7660.516456] Lustre: mylustre-OST0001: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:33) [ 7663.900926] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7671.115129] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7674.711122] Lustre: 244094:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7685.355665] Lustre: server umount mylustre-MDT0000 complete [ 7686.117943] Lustre: mylustre-MDT0000-lwp-OST0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7686.126827] LustreError: 241077:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7686.152924] LustreError: 241077:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7688.633639] LustreError: 241057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837932 with bad export cookie 7407167350910779261 [ 7688.649484] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7688.651255] LustreError: 241057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7691.158484] Lustre: server umount mylustre-MDT0001 complete [ 7705.003159] Lustre: server umount mylustre-OST0000 complete [ 7707.107780] Lustre: 234931:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788837934/real 1788837934] req@ffff99d6c3cac700 x1875732270826368/t0(0) o400->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788837950 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7707.137230] Lustre: mylustre-MDT0001-lwp-OST0001: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7707.143249] Lustre: Skipped 3 previous similar messages [ 7708.922852] Lustre: server umount mylustre-OST0001 complete [ 7729.055508] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 7738.518134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7738.868258] LustreError: 246402:0:(ldlm_lib.c:1190:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7738.950051] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7743.612191] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7753.562457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7758.521589] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7761.667563] Lustre: 247543:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7768.719642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7776.330984] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7778.504179] Lustre: tfs-OST0000: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7783.625310] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49792) [ 7786.082386] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7791.624330] Lustre: tfs-OST0001: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:65) [ 7793.137979] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7800.809296] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7804.568770] Lustre: 249418:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7815.140174] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7815.149740] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7815.180602] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 7817.188593] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 7817.209649] Lustre: Skipped 3 previous similar messages [ 7818.891838] Lustre: server umount tfs-MDT0000 complete [ 7822.310745] LustreError: 249184:0:(ldlm_lib.c:1190:target_handle_connect()) tfs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7822.342199] LustreError: 249184:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 7822.697539] LustreError: 247546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838066 with bad export cookie 7407167350910781438 [ 7822.701672] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7822.707915] LustreError: 247546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7822.920191] Lustre: server umount tfs-MDT0001 complete [ 7836.807635] Lustre: server umount tfs-OST0000 complete [ 7851.374575] Lustre: server umount tfs-OST0001 complete [ 7871.677616] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 7882.471671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7882.992736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7882.999964] Lustre: Skipped 3 previous similar messages [ 7887.726503] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7897.035282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7901.779556] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7905.178785] Lustre: 252811:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7913.259263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7920.827901] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7922.966521] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 7928.133302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49824) [ 7929.042751] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7934.491769] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:97) [ 7935.969376] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7943.675585] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7947.999368] Lustre: 254681:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7955.373032] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 23:29:57 (1788838197) [ 7974.376883] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7974.385238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7974.411766] Lustre: Skipped 3 previous similar messages [ 7974.424214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7980.522989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7980.529983] Lustre: Skipped 5 previous similar messages [ 7986.144143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7986.323644] Lustre: server umount lustre-MDT0000 complete [ 7990.413389] LustreError: 253189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838234 with bad export cookie 7407167350910783356 [ 7990.430771] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7990.432526] LustreError: 253189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7990.753643] LustreError: 251668:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7990.759919] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7990.773924] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7990.773936] Lustre: Skipped 6 previous similar messages [ 7990.792937] LustreError: 251668:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 7990.838836] Lustre: Skipped 4 previous similar messages [ 7996.860888] Lustre: server umount lustre-MDT0001 complete [ 8006.647738] Lustre: server umount lustre-OST0000 complete [ 8016.550220] Lustre: server umount lustre-OST0001 complete [ 8023.482261] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [ 8033.780207] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 8072.962577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8073.312603] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8073.365268] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8073.494245] Lustre: lustre-MDT0000: new disk, initializing [ 8073.578562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 188-180 [ 8073.584753] Lustre: Skipped 3 previous similar messages [ 8073.612731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8078.088957] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 8088.269641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8088.460980] Lustre: 257869:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8088.566795] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8088.570984] Lustre: Skipped 1 previous similar message [ 8088.681506] Lustre: lustre-MDT0001: new disk, initializing [ 8088.917333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8088.931459] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8093.848415] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 8103.744728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8104.054121] Lustre: lustre-OST0000: new disk, initializing [ 8104.060669] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8104.067545] Lustre: 259450:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8105.323867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8105.343635] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8105.501150] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8111.016693] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 8124.784528] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8125.004950] Lustre: lustre-OST0001: new disk, initializing [ 8125.009357] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8125.020771] Lustre: 260321:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8126.215310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8126.247730] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8126.395586] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8132.853893] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 8147.335223] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 23:33:09 (1788838389) [ 8163.877795] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 23:33:25 (1788838405) [ 8182.242787] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8182.258791] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8182.264621] LustreError: Skipped 1 previous similar message [ 8182.270533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8182.277829] Lustre: Skipped 1 previous similar message [ 8182.300708] Lustre: Skipped 4 previous similar messages [ 8185.845703] Lustre: server umount lustre-OST0000 complete [ 8191.457548] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8197.263801] Lustre: server umount lustre-MDT0000 complete [ 8201.436963] LustreError: 257860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838445 with bad export cookie 7407167350910785134 [ 8201.442164] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8201.449214] LustreError: 257860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8207.846626] Lustre: server umount lustre-MDT0001 complete [ 8222.993956] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 8297.699223] Lustre: server umount lustre-OST0001 complete [ 8305.678242] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [ 8325.274303] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 8366.241820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8366.455883] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8366.479821] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8366.556660] Lustre: lustre-MDT0000: new disk, initializing [ 8366.710447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8366.717711] Lustre: Skipped 3 previous similar messages [ 8366.744191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8371.452701] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 8383.739206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8383.860687] Lustre: 265178:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8383.906375] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8383.909130] Lustre: Skipped 1 previous similar message [ 8383.963597] Lustre: lustre-MDT0001: new disk, initializing [ 8384.101920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8384.115325] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8390.497856] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 8401.072683] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8407.155904] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8414.018106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8414.284201] Lustre: 267151:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8415.628345] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8415.788778] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8420.351398] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 8430.284571] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8436.197539] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8436.200828] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8436.221820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8436.221832] Lustre: Skipped 6 previous similar messages [ 8436.258353] Lustre: Skipped 5 previous similar messages [ 8442.116905] Lustre: server umount lustre-OST0000 complete [ 8453.264045] Lustre: server umount lustre-MDT0000 complete [ 8456.674315] LustreError: 265190:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8456.691898] LustreError: 265190:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 8457.604445] LustreError: 265984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838701 with bad export cookie 7407167350910786604 [ 8457.612860] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8457.616272] LustreError: 265984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8458.472719] Lustre: server umount lustre-MDT0001 complete [ 8474.625594] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 8479.580972] Key type lgssc unregistered [ 8480.081744] LNet: 268910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8480.094903] LNetError: 268910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8481.127260] LNet: Removed LNI 192.168.202.148@tcp [ 8482.547379] Key type .llcrypt unregistered [ 8482.558748] Key type ._llcrypt unregistered [ 8522.021811] Key type ._llcrypt registered [ 8522.027617] Key type .llcrypt registered [ 8522.275243] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 8524.008244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8524.101840] alg: No test for adler32 (adler32-zlib) [ 8525.250818] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 8525.550226] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 8527.226482] Key type lgssc registered [ 8528.361687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8537.007890] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8537.021440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8538.425441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8543.213511] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 8548.840277] LustreError: 270371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8548.855363] LustreError: 270371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8551.928617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8552.377948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8557.654985] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 8565.565096] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8572.179684] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8579.352203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8579.732775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8583.866377] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 8587.290313] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 8595.193475] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8596.049945] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 8599.521791] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8599.530410] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8599.547077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8602.593302] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8602.599568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8602.600834] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8602.629663] Lustre: Skipped 1 previous similar message [ 8604.098391] Lustre: server umount lustre-OST0000 complete [ 8608.737631] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8608.747596] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8608.762929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8612.834673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8612.845959] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8614.523628] Lustre: server umount lustre-MDT0000 complete [ 8617.763279] LustreError: 270353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838861 with bad export cookie 8006561154081978871 [ 8617.765095] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8617.773432] LustreError: 270353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8618.068134] Lustre: server umount lustre-MDT0001 complete [ 8629.773522] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 8632.454435] Key type lgssc unregistered [ 8632.737818] LNet: 273955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8632.749884] LNetError: 273955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8632.761067] LNet: Removed LNI 192.168.202.148@tcp [ 8633.600229] Key type .llcrypt unregistered [ 8633.601887] Key type ._llcrypt unregistered [ 8650.508604] Key type ._llcrypt registered [ 8650.511726] Key type .llcrypt registered [ 8650.608654] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 8651.578693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8651.665711] alg: No test for adler32 (adler32-zlib) [ 8652.666617] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 8652.878637] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 8654.520179] Key type lgssc registered [ 8655.480587] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8662.858316] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8662.876270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8678.368589] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8683.488728] LustreError: 275147:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.148@tcp: failed processing log, type 4: rc = -110 [ 8710.112241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8716.016865] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 8723.114374] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8725.352807] Lustre: server umount lustre-OST0000 complete [ 8741.288420] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 8744.042555] Key type lgssc unregistered [ 8744.327973] LNet: 276776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8744.333308] LNetError: 276776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8744.353867] LNet: Removed LNI 192.168.202.148@tcp [ 8745.198551] Key type .llcrypt unregistered [ 8745.200417] Key type ._llcrypt unregistered [ 8782.384451] Key type ._llcrypt registered [ 8782.386284] Key type .llcrypt registered [ 8782.466319] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 8783.526826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8783.550530] alg: No test for adler32 (adler32-zlib) [ 8784.549757] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 8784.773679] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 8786.416315] Key type lgssc registered [ 8787.347796] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8799.090800] loop0: detected capacity change from 0 to 204800000 [ 8799.242503] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8799.257926] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8799.579944] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8800.622107] loop0: detected capacity change from 0 to 204800000 [ 8800.833487] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8800.844053] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8801.235879] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8802.316528] loop0: detected capacity change from 0 to 204800000 [ 8802.562876] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8802.574637] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8802.729667] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8803.849612] loop0: detected capacity change from 0 to 204800000 [ 8804.064386] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8805.465547] loop0: detected capacity change from 0 to 204800000 [ 8805.507198] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8809.115692] loop0: detected capacity change from 0 to 204800000 [ 8809.147622] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8812.428607] loop0: detected capacity change from 0 to 204800000 [ 8812.458410] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8815.789602] loop0: detected capacity change from 0 to 204800000 [ 8815.802625] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8820.085525] loop0: detected capacity change from 0 to 204800000 [ 8820.166914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8820.219649] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8820.283934] Lustre: 280181:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8820.291419] Lustre: 280181:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 8820.298453] Lustre: 280181:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8820.306664] Lustre: 280181:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8820.314051] Lustre: 280181:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8820.319612] Lustre: 280181:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 8820.325346] Lustre: 280181:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 8820.336404] Lustre: 280181:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 8820.345732] Lustre: 280181:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 8820.360427] Lustre: 280181:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 8820.367866] Lustre: 280181:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 8820.377540] Lustre: 280181:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 8820.386367] Lustre: 280181:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 8820.394456] Lustre: 280181:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 8826.510889] Lustre: server umount MGS complete [ 8827.360535] loop0: detected capacity change from 0 to 204800000 [ 8827.494148] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8828.084521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8828.139121] Lustre: lustre-MDT0000: Aborting client recovery [ 8829.119491] loop1: detected capacity change from 0 to 204800000 [ 8829.271675] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8829.381813] Lustre: 280614:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8829.408676] Lustre: 280614:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 8829.430427] Lustre: 280614:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 8829.443672] Lustre: 280614:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8829.450956] Lustre: 280614:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8829.462525] Lustre: 280614:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 8829.482922] Lustre: 280614:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8829.517431] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8829.698916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8829.753619] Lustre: lustre-MDT0001: Aborting client recovery [ 8830.560772] loop2: detected capacity change from 0 to 204800000 [ 8830.760298] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8830.855557] Lustre: 280693:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8830.866619] Lustre: 280693:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8830.881864] Lustre: 280693:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8830.891441] Lustre: 280693:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8830.901022] Lustre: 280693:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8830.915280] Lustre: 280693:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8830.922181] Lustre: 280693:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8831.020973] Lustre: Found index 0 for lustre-OST0000, updating log [ 8831.049434] Lustre: 280711:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8831.106277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8831.122579] Lustre: lustre-OST0000: Aborting recovery [ 8832.154375] loop3: detected capacity change from 0 to 204800000 [ 8832.375784] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8832.471146] Lustre: 280776:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8832.476838] Lustre: 280776:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8832.489405] Lustre: 280776:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8832.499371] Lustre: 280776:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8832.510970] Lustre: 280776:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8832.522280] Lustre: 280776:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8832.531269] Lustre: 280776:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8832.563492] Lustre: Found index 1 for lustre-OST0001, updating log [ 8832.621615] Lustre: 280782:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8832.750492] Lustre: lustre-OST0001: Aborting recovery [ 8837.864144] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8837.870319] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8837.908719] LustreError: 280558:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 8837.919340] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 8837.928676] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 8839.911754] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 8839.918334] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 8840.940119] LustreError: 280525:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 5796us: rc = -2 [ 8841.442478] LustreError: 280523:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 596us: rc = -2 [ 8841.455717] LustreError: 280523:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 30 previous similar messages [ 8855.521341] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8855.531337] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8855.536788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8855.543953] Lustre: Skipped 3 previous similar messages [ 8855.549717] Lustre: Skipped 2 previous similar messages [ 8860.641645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8860.645288] Lustre: Skipped 2 previous similar messages [ 8865.762166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8865.771472] Lustre: Skipped 3 previous similar messages [ 8868.320256] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8868.605517] Lustre: server umount lustre-MDT0000 complete [ 8869.571104] LustreError: 280508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839113 with bad export cookie 15581481949489576183 [ 8869.598053] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8870.880745] LustreError: 280522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8870.882980] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8870.883544] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8870.912282] LustreError: 280522:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8870.930313] Lustre: Skipped 1 previous similar message [ 8874.913969] LustreError: 280523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8874.914374] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8874.933393] Lustre: Skipped 2 previous similar messages [ 8876.080584] Lustre: server umount lustre-MDT0001 complete [ 8891.363822] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8891.524338] Lustre: server umount lustre-OST0000 complete [ 8898.573157] Lustre: server umount lustre-OST0001 complete [ 8905.812699] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 23:45:47 (1788839147) [ 8933.542391] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [ 8943.580100] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 8988.324571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8988.616251] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8988.671254] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8988.794637] Lustre: lustre-MDT0000: new disk, initializing [ 8988.881499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8988.888963] Lustre: Skipped 1 previous similar message [ 8988.898942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8993.985469] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 9006.645919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9006.845990] Lustre: 284855:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9006.950477] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9006.961043] Lustre: Skipped 1 previous similar message [ 9007.104833] Lustre: lustre-MDT0001: new disk, initializing [ 9007.292401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9007.346870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9007.358814] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9012.454671] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 9024.112915] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9029.985318] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9037.508197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9037.959140] Lustre: lustre-OST0000: new disk, initializing [ 9037.962701] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9037.969577] Lustre: 286821:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9038.047361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9043.551890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9043.567200] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9043.828591] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9046.635248] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 9059.019695] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9066.352315] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9080.204165] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9091.431947] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9099.481047] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 9141.218734] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9141.227786] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9141.238377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9147.514171] Lustre: server umount lustre-OST0000 complete [ 9151.466228] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9151.492996] Lustre: Skipped 2 previous similar messages [ 9157.709374] Lustre: server umount lustre-MDT0000 complete [ 9161.694782] LustreError: 284846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839405 with bad export cookie 15581481949489578906 [ 9161.720332] LustreError: 284846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9161.726055] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9162.035554] Lustre: server umount lustre-MDT0001 complete [ 9170.955220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9176.342387] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 9182.316828] Lustre: server umount MGS complete [ 9194.360436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9195.019721] LustreError: 290841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9195.038368] LustreError: 290841:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9195.044344] LustreError: 290854:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9195.109298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9199.698522] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 9200.097919] LustreError: 290842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9205.232816] LustreError: 290841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9208.375159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9213.334738] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 9220.572973] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9225.600322] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9232.610777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9232.911706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9232.919135] Lustre: Skipped 1 previous similar message [ 9239.024977] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 9247.091881] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9279.969603] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9279.974018] LustreError: Skipped 1 previous similar message [ 9279.976566] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9279.993196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9279.997817] Lustre: Skipped 7 previous similar messages [ 9293.285744] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9293.691433] Lustre: server umount lustre-OST0000 complete [ 9299.428626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9299.447887] Lustre: Skipped 2 previous similar messages [ 9304.546197] LustreError: 290841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9304.770236] Lustre: server umount lustre-MDT0000 complete [ 9308.564060] LustreError: 290823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839552 with bad export cookie 15581481949489580859 [ 9308.570928] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9308.574952] LustreError: 290823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9309.135187] Lustre: server umount lustre-MDT0001 complete [ 9321.221622] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 9324.734668] Key type lgssc unregistered [ 9325.121891] LNet: 294789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9325.128321] LNetError: 294789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9325.144792] LNet: Removed LNI 192.168.202.148@tcp [ 9326.214735] Key type .llcrypt unregistered [ 9326.227601] Key type ._llcrypt unregistered [ 9365.619039] Key type ._llcrypt registered [ 9365.620701] Key type .llcrypt registered [ 9365.729547] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [ 9380.834495] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 9381.724675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9381.775366] alg: No test for adler32 (adler32-zlib) [ 9382.933543] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 9383.141441] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 9384.864223] Key type lgssc registered [ 9386.099853] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9421.222323] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9421.247783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9422.524586] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9422.549097] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9422.649788] Lustre: lustre-MDT0000: new disk, initializing [ 9422.781821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9422.807742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9427.078285] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 9438.578103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9438.782384] Lustre: 298746:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9438.858613] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9438.862281] Lustre: Skipped 1 previous similar message [ 9438.947600] Lustre: lustre-MDT0001: new disk, initializing [ 9439.060477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9439.120295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9439.129754] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9443.475333] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 9453.543429] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9458.896189] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9464.810631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9465.061342] Lustre: lustre-OST0000: new disk, initializing [ 9465.067606] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9465.078464] Lustre: 300712:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9465.171747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9470.535322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9470.543350] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9470.683182] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9471.184532] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 9480.112375] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9487.827773] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9496.984874] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9502.783951] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9552.882905] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9552.883401] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9552.891900] Lustre: Skipped 1 previous similar message [ 9552.892051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9554.677632] Lustre: server umount lustre-OST0000 complete [ 9562.081156] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9562.096362] LustreError: Skipped 1 previous similar message [ 9562.108065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9562.136260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9562.146676] Lustre: Skipped 1 previous similar message [ 9564.493997] Lustre: server umount lustre-MDT0000 complete [ 9568.060682] LustreError: 301567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839811 with bad export cookie 10767736406159464344 [ 9568.074257] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9568.096519] LustreError: 301567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9568.376216] Lustre: server umount lustre-MDT0001 complete [ 9576.392937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9581.232038] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 9585.875672] Lustre: server umount MGS complete [ 9597.173290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9597.541445] LustreError: 304541:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9597.613146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9602.007508] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 9603.044650] LustreError: 304528:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9603.078106] LustreError: 304528:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9608.162333] LustreError: 304529:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9610.253087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9610.564211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9615.119460] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 9622.139608] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9628.663120] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9634.576610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9634.819790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9641.068559] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 9647.977300] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9675.956152] Lustre: server umount lustre-OST0000 complete [ 9682.402721] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9682.411686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9682.425951] Lustre: Skipped 1 previous similar message [ 9682.434116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9682.436788] Lustre: Skipped 2 previous similar messages [ 9689.058220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9689.072550] Lustre: Skipped 3 previous similar messages [ 9689.161913] Lustre: server umount lustre-MDT0000 complete [ 9692.651539] Lustre: 296971:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788839920/real 1788839920] req@0000000072fae96c x1875734299814016/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788839936 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9692.691231] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9692.723242] Lustre: Skipped 1 previous similar message [ 9692.734766] LustreError: 304529:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9692.800393] LustreError: 304529:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9693.776426] LustreError: 304510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839937 with bad export cookie 10767736406159466206 [ 9693.777658] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9693.784275] LustreError: 304510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9694.384178] Lustre: server umount lustre-MDT0001 complete [ 9709.151843] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 9714.458163] Key type lgssc unregistered [ 9714.925066] LNet: 308473:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9714.931631] LNetError: 308473:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9714.970065] LNet: Removed LNI 192.168.202.148@tcp [ 9716.330590] Key type .llcrypt unregistered [ 9716.333488] Key type ._llcrypt unregistered [ 9739.189425] Key type ._llcrypt registered [ 9739.191937] Key type .llcrypt registered [ 9739.323872] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [ 9753.420019] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 9754.381848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9754.425285] alg: No test for adler32 (adler32-zlib) [ 9755.601221] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 9755.975185] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 9757.704386] Key type lgssc registered [ 9759.088132] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9781.842195] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 9796.527305] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9796.539560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9797.680795] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9797.706295] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9797.814225] Lustre: lustre-MDT0000: new disk, initializing [ 9797.910792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9797.931223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9801.991125] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 9813.766719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9814.088927] Lustre: lustre-OST0000: new disk, initializing [ 9814.092570] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9814.097240] Lustre: Skipped 1 previous similar message [ 9814.111415] Lustre: 312520:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9814.222161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9822.269854] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 9822.340397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9822.347657] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9822.390434] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10579.952900] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10579.961182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10580.825794] Lustre: server umount lustre-MDT0000 complete [10584.635549] LustreError: 311709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840828 with bad export cookie 8015934283142263410 [10584.641976] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10584.649945] LustreError: 311709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10585.013926] Lustre: server umount lustre-OST0000 complete [10612.510550] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [10624.562807] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [10660.765871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10660.965113] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10660.985306] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10661.048241] Lustre: lustre-MDT0000: new disk, initializing [10661.098682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10661.110957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10666.076921] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [10677.270481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10677.463060] Lustre: 316857:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10677.559436] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10677.564148] Lustre: Skipped 1 previous similar message [10677.679965] Lustre: lustre-MDT0001: new disk, initializing [10677.787482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10677.814647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10677.828901] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10682.575205] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [10693.128669] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10698.843861] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10706.351977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10706.737879] Lustre: lustre-OST0000: new disk, initializing [10706.744550] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10706.753636] Lustre: 318822:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10706.900136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10713.498740] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [10714.694709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10714.709625] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10714.832840] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10723.515843] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10728.005645] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10728.277885] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10733.305589] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10733.612252] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10740.712329] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10740.718804] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10740.750275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10740.761444] Lustre: Skipped 1 previous similar message [10742.054574] Lustre: server umount lustre-OST0000 complete [10749.422286] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10749.441156] LustreError: Skipped 1 previous similar message [10749.443581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10749.451723] Lustre: Skipped 1 previous similar message [10749.456258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10752.237356] Lustre: server umount lustre-MDT0000 complete [10755.540353] LustreError: 316847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840999 with bad export cookie 8015934283145204222 [10755.558142] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10756.065936] Lustre: server umount lustre-MDT0001 complete [10759.516986] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [10761.111351] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 00:16:42 (1788841002) [10767.815392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10768.235352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10772.658751] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [10773.489672] LustreError: 321135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10773.509549] LustreError: 321135:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10777.589464] LustreError: 321136:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10781.141279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10781.548992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10785.966673] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [10793.107993] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10798.308460] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10804.381244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10804.826512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10811.530143] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [10818.817443] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10827.099160] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10827.335499] Lustre: lustre-OST0001: new disk, initializing [10827.344746] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10827.355957] Lustre: 323970:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10827.464916] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10833.980736] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [10837.143857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10837.161976] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10837.270528] LustreError: 324430:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10838.313962] LustreError: 324430:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10838.350297] LustreError: 324430:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [10839.400861] LustreError: 324438:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10839.412126] LustreError: 324438:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [10841.450126] LustreError: 324430:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10841.461601] LustreError: 324430:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [10845.156282] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10845.540493] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10845.708846] LustreError: 324430:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10845.735925] LustreError: 324430:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 8 previous similar messages [10853.161295] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9199fbacf800.ost_server_uuid 50 [10853.937800] LustreError: 324438:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10853.947381] LustreError: 324438:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 14 previous similar messages [10855.332621] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9199fbacf800.ost_server_uuid in FULL state after 0 sec [10861.872404] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9199fbacf800.ost_server_uuid 50 [10863.345767] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9199fbacf800.ost_server_uuid in IDLE state after 0 sec [10865.288124] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10880.993465] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10880.996664] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10881.000265] LustreError: Skipped 1 previous similar message [10881.004681] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10881.012742] Lustre: Skipped 2 previous similar messages [10881.027478] Lustre: Skipped 3 previous similar messages [10886.174744] Lustre: lustre-OST0001: Not available for connect from 192.168.202.48@tcp (stopping) [10886.177649] Lustre: Skipped 3 previous similar messages [10893.280284] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10893.413189] Lustre: server umount lustre-OST0001 complete [10893.811318] LustreError: 322956:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10893.837082] LustreError: 322956:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10896.426981] LustreError: 322957:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10900.450971] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10900.456991] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10900.467863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10900.472187] Lustre: Skipped 3 previous similar messages [10904.047484] LustreError: 322956:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10904.075618] LustreError: 322956:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10905.980506] Lustre: server umount lustre-OST0000 complete [10914.276584] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10914.287924] Lustre: Skipped 2 previous similar messages [10916.179777] Lustre: server umount lustre-MDT0000 complete [10919.396380] LustreError: 321135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10919.425819] LustreError: 321135:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10919.955238] LustreError: 323968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841163 with bad export cookie 8015934283145205104 [10919.955941] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10919.964402] LustreError: 323968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10920.258462] Lustre: server umount lustre-MDT0001 complete [10929.830975] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [10932.328659] Key type lgssc unregistered [10932.643765] LNet: 326658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10932.649184] LNetError: 326658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10932.687601] LNet: Removed LNI 192.168.202.148@tcp [10933.507250] Key type .llcrypt unregistered [10933.511682] Key type ._llcrypt unregistered [10956.129606] Key type ._llcrypt registered [10956.132569] Key type .llcrypt registered [10956.226980] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [10957.720855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10957.825151] alg: No test for adler32 (adler32-zlib) [10958.797504] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [10959.075351] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [10960.784518] Key type lgssc registered [10962.034381] Lustre: Echo OBD driver; http://www.lustre.org/ [10969.509187] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10969.522897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10971.016421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10976.293880] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [10980.331355] LustreError: 328082:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10980.350968] LustreError: 328082:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10983.891402] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10985.442381] LustreError: 328083:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10990.563183] LustreError: 328082:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10990.782187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10991.329281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10997.745228] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [11006.573319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11006.897951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11013.201368] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [11016.133411] LustreError: 329762:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11016.160802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [11020.090711] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11021.305142] LustreError: 329760:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11021.339322] LustreError: 329760:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11026.289253] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11026.448377] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11031.565894] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [11032.704922] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [11039.676928] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11047.287721] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [11047.522273] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11149.292780] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11149.300839] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11149.328311] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11153.444452] Lustre: server umount lustre-OST0001 complete [11154.439270] LustreError: 329762:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11154.484204] LustreError: 329762:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11164.642510] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11164.650136] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11164.658284] LustreError: Skipped 1 previous similar message [11164.671129] Lustre: Skipped 1 previous similar message [11164.671914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11164.671918] Lustre: Skipped 1 previous similar message [11167.526653] Lustre: server umount lustre-OST0000 complete [11174.885104] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11174.889954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11174.902503] Lustre: Skipped 2 previous similar messages [11174.912751] Lustre: Skipped 2 previous similar messages [11178.762803] Lustre: server umount lustre-MDT0000 complete [11180.009099] LustreError: 328082:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11180.026960] LustreError: 328082:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11182.482781] LustreError: 328063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841426 with bad export cookie 2616841162776361468 [11182.507199] LustreError: 328063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11182.507630] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11182.873160] Lustre: server umount lustre-MDT0001 complete [11194.208924] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [11197.349438] Key type lgssc unregistered [11197.733171] LNet: 333025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11197.743531] LNetError: 333025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11198.764210] LNet: Removed LNI 192.168.202.148@tcp [11199.811384] Key type .llcrypt unregistered [11199.813353] Key type ._llcrypt unregistered [11250.322710] Key type ._llcrypt registered [11250.329890] Key type .llcrypt registered [11250.432749] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [11251.565197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11251.662837] alg: No test for adler32 (adler32-zlib) [11253.120377] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [11253.487443] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [11255.304354] Key type lgssc registered [11256.643654] Lustre: Echo OBD driver; http://www.lustre.org/ [11267.481704] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11267.502040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11269.268192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11274.395772] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [11278.845289] LustreError: 335326:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11278.883839] LustreError: 335326:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11283.939725] LustreError: 335327:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11286.803495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11287.595138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11291.629747] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [11295.403135] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11303.965788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11304.502544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11311.485518] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [11312.699614] LustreError: 336818:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11313.718773] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [11318.759188] LustreError: 336819:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11318.777388] LustreError: 336819:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11321.053489] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11321.265849] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11326.481392] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [11329.220727] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [11338.582735] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11346.944558] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11362.277978] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11362.281400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11362.291487] Lustre: Skipped 2 previous similar messages [11362.302145] Lustre: Skipped 3 previous similar messages [11367.401589] LustreError: 337508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11367.421881] LustreError: 337508:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11368.130651] Lustre: server umount lustre-MDT0000 complete [11372.516138] LustreError: 335308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841616 with bad export cookie 8875323367592536811 [11372.536615] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11372.544053] LustreError: 335308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11372.568931] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11372.588986] Lustre: Skipped 1 previous similar message [11372.595424] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11372.600935] Lustre: Skipped 1 previous similar message [11372.893345] Lustre: server umount lustre-MDT0001 complete [11379.559150] Lustre: server umount lustre-OST0000 complete [11384.450624] Lustre: server umount lustre-OST0001 complete [11402.043973] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [11412.694074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11413.396792] LustreError: 340593:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11413.441453] LustreError: 340593:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11413.516825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11419.237685] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [11430.472392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11430.854973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11436.263268] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [11439.647681] Lustre: 341734:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11448.057281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11449.045626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11456.236931] LustreError: 342087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11456.249468] LustreError: 342087:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11458.316532] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [11459.332115] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [11471.347569] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11476.364067] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [11477.008341] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [11484.149794] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11488.787226] Lustre: 343605:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11497.449869] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11497.451756] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11497.468324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11497.468646] Lustre: Skipped 3 previous similar messages [11499.152057] Lustre: server umount lustre-MDT0000 complete [11502.574542] LustreError: 340590:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11502.631319] LustreError: 340590:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [11503.685370] LustreError: 340573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841747 with bad export cookie 8875323367592538512 [11503.686848] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11503.693330] LustreError: 340573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11503.992878] Lustre: server umount lustre-MDT0001 complete [11519.046660] Lustre: server umount lustre-OST0000 complete [11534.266780] Lustre: server umount lustre-OST0001 complete [11538.348525] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [11539.895098] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [11542.428516] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 00:29:43 (1788841783) [11565.516369] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [11576.684063] loop0: detected capacity change from 0 to 1099511627776 [11577.566612] SGI XFS with ACLs, security attributes, quota, no debug enabled [11577.576582] *** VALIDATE xfs *** [11577.598164] XFS (loop0): Mounting V5 Filesystem [11577.623978] XFS (loop0): Ending clean mount [11579.633800] loop1: detected capacity change from 0 to 18691697672192 [11724.487235] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [11735.030830] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [11774.763056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11775.058783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11775.085279] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11775.158585] Lustre: lustre-MDT0000: new disk, initializing [11775.245866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11775.253843] Lustre: Skipped 1 previous similar message [11775.269804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11780.245747] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [11790.486236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11790.600445] Lustre: 348749:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11790.654700] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11790.658196] Lustre: Skipped 1 previous similar message [11790.717661] Lustre: lustre-MDT0001: new disk, initializing [11790.842563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11790.855266] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11796.240660] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [11806.322121] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11812.822687] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11820.854167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11821.148680] Lustre: lustre-OST0000: new disk, initializing [11821.153621] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11821.158804] Lustre: 350720:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11822.748706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11822.771633] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11822.944814] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11830.721767] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [11842.630107] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11847.784298] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11848.095824] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11853.074982] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11853.299234] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11858.921564] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11858.928149] Lustre: Skipped 1 previous similar message [11858.933390] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11858.934856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11858.953658] Lustre: Skipped 4 previous similar messages [11861.845605] Lustre: server umount lustre-OST0000 complete [11867.618385] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11867.625068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11867.644105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11872.306931] Lustre: server umount lustre-MDT0000 complete [11874.275447] LustreError: 348762:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11874.297392] LustreError: 348762:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11877.040376] LustreError: 348743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842120 with bad export cookie 8875323367592540066 [11877.051274] LustreError: 348743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11877.055615] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11877.442351] Lustre: server umount lustre-MDT0001 complete [11886.535762] XFS (loop0): Unmounting Filesystem [11891.761902] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 00:35:33 (1788842133) [11899.736528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11900.517774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11900.529639] Lustre: Skipped 2 previous similar messages [11906.571536] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [11916.963312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11923.422326] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [11935.306263] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11941.500155] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11949.328522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11957.851650] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [11965.707776] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11981.797552] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11981.799153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11981.803066] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11981.813840] Lustre: Skipped 2 previous similar messages [11981.838310] Lustre: Skipped 3 previous similar messages [11988.082961] Lustre: server umount lustre-OST0000 complete [11998.060538] Lustre: server umount lustre-MDT0000 complete [12001.546927] LustreError: 353264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842245 with bad export cookie 8875323367592540948 [12001.571051] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12002.539586] Lustre: server umount lustre-MDT0001 complete [12016.960576] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [12021.550257] Key type lgssc unregistered [12022.038676] LNet: 356914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12022.053407] LNetError: 356914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12022.087385] LNet: Removed LNI 192.168.202.148@tcp [12023.682354] Key type .llcrypt unregistered [12023.688337] Key type ._llcrypt unregistered [12055.093647] Key type ._llcrypt registered [12055.103784] Key type .llcrypt registered [12055.214855] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [12056.365531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12056.408601] alg: No test for adler32 (adler32-zlib) [12057.578490] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [12057.852232] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [12059.505475] Key type lgssc registered [12060.808469] Lustre: Echo OBD driver; http://www.lustre.org/ [12068.900377] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12068.929896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12070.299033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12075.238855] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [12079.595732] LustreError: 358480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12079.617487] LustreError: 358480:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12084.715217] LustreError: 358479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12085.402720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12085.965486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12091.426498] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [12099.976330] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12106.127136] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12114.690254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12114.851863] LustreError: 360298:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12115.217202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12123.265528] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [12133.493475] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12142.561353] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12142.571958] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12142.581245] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12145.754052] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.48@tcp (stopping) [12146.150619] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12150.821118] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.48@tcp (stopping) [12150.826765] Lustre: Skipped 2 previous similar messages [12155.872529] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12155.937847] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.48@tcp (stopping) [12155.953952] Lustre: Skipped 2 previous similar messages [12156.153372] Lustre: server umount lustre-MDT0001 complete [12156.394679] LustreError: 358479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12156.410269] LustreError: 358479:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12161.066512] LustreError: 358475:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12167.220471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12167.364698] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12167.374358] Lustre: Found index 1 for lustre-MDT0001, updating log [12167.448794] Lustre: 358468:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12167.731695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12172.162718] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [12172.794756] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12174.857462] LustreError: 362043:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12174.867363] LustreError: 362043:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12174.880877] LustreError: 358471:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12174.967435] LustreError: 362050:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.148@tcp: cfg command failed: rc = -17 [12174.976387] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [12174.976387] [12186.677330] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.48@tcp (stopping) [12188.131601] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12188.134165] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12188.167116] Lustre: Skipped 1 previous similar message [12190.016895] Lustre: server umount lustre-MDT0001 complete [12191.771538] LustreError: 358474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12191.787825] LustreError: 358474:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12199.777222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12199.944642] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12199.951943] Lustre: Found index 1 for lustre-MDT0001, updating log [12200.011434] Lustre: 358469:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12200.301361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12204.777174] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [12205.572615] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12208.159272] LustreError: 363155:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12208.170516] LustreError: 363155:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12208.178052] LustreError: 358471:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12208.185950] LustreError: 358471:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12208.272677] LustreError: 363158:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [12208.285950] LustreError: 363158:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [12208.292177] LustreError: 363158:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.148@tcp: cfg command failed: rc = -17 [12208.302103] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.202.148@tcp [12208.302103] [12512.738827] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12512.761873] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12512.768365] Lustre: Skipped 2 previous similar messages [12517.685232] Lustre: server umount lustre-MDT0001 complete [12517.867593] LustreError: 358474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12517.900177] LustreError: 358474:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [12528.166756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12528.286273] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12528.297607] Lustre: Found index 1 for lustre-MDT0001, updating log [12528.352482] Lustre: 358467:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12528.735231] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12533.754943] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12533.788654] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [12535.816306] LustreError: 371736:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12535.824294] LustreError: 371736:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12535.829872] LustreError: 358471:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12535.837619] LustreError: 358471:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12535.930771] LustreError: 371745:0:(lod_lov.c:208:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [12535.965663] LustreError: 371745:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.148@tcp: cfg command failed: rc = -114 [12535.981798] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [12535.981798] [12856.291978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12856.297305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12856.308872] Lustre: Skipped 1 previous similar message [12856.312790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12856.321485] Lustre: Skipped 1 previous similar message [12862.247512] Lustre: server umount lustre-MDT0000 complete [12862.446630] LustreError: 358476:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12862.468194] LustreError: 358476:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [12865.847797] LustreError: 360321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843109 with bad export cookie 6861620715751362803 [12865.850598] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12865.855668] LustreError: 360321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12867.553513] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12867.566084] Lustre: Skipped 2 previous similar messages [12880.354033] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12880.683738] Lustre: server umount lustre-MDT0001 complete [12885.092780] Lustre: server umount lustre-OST0000 complete [12897.063858] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [12909.800410] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [12962.136801] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 00:53:23 (1788843203) [12971.239165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12971.678676] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12971.766195] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12971.985898] Lustre: lustre-MDT0000: new disk, initializing [12972.192410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12972.230463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12977.777212] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [12989.500783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12989.582181] Lustre: 382168:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12989.619280] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12989.622199] Lustre: Skipped 1 previous similar message [12989.666188] Lustre: lustre-MDT0001: new disk, initializing [12989.735389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12989.768774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12989.775512] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12996.169294] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [13006.384407] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13012.425703] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13019.955203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13020.378732] Lustre: lustre-OST0000: new disk, initializing [13020.383530] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13020.393161] Lustre: 384138:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13020.527816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13025.865755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13025.887932] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13026.000964] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13029.119101] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [13039.193940] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13048.398615] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13048.414353] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13048.458538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13048.473075] Lustre: Skipped 6 previous similar messages [13048.754909] Lustre: server umount lustre-MDT0000 complete [13051.876269] LustreError: 382179:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13051.897149] LustreError: 382179:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13053.162949] LustreError: 382975:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [13053.166021] LustreError: 382963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843296 with bad export cookie 6861620715751364756 [13053.168694] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13053.230158] LustreError: 382963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13053.546165] Lustre: server umount lustre-MDT0001 complete [13068.252696] Lustre: server umount lustre-OST0000 complete [13105.126836] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [13118.361545] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [13156.797837] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12864 sec ========= 00:56:38 (1788843398) [13158.740343] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 00:56:40 (1788843400) === [13162.789634] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 00:56:44 (1788843404) === [13213.799893] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [13217.462448] Key type lgssc unregistered [13217.750665] LNet: 390507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13217.768534] LNetError: 390507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13218.791544] LNet: Removed LNI 192.168.202.148@tcp [13220.042811] Key type .llcrypt unregistered [13220.050959] Key type ._llcrypt unregistered