[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 859601936 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003221] x2apic enabled [ 0.004006] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.007812] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009009] pid_max: default: 32768 minimum: 301 [ 0.010133] LSM: Security Framework initializing [ 0.011040] Yama: becoming mindful. [ 0.012027] SELinux: Initializing. [ 0.014033] *** VALIDATE selinux *** [ 0.021485] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026258] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027152] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028124] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030065] *** VALIDATE tmpfs *** [ 0.031025] *** VALIDATE proc *** [ 0.032259] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.035105] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036170] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038030] Spectre V2 : User space: Vulnerable [ 0.039007] Speculative Store Bypass: Vulnerable [ 0.042652] debug: unmapping init [mem 0xffffffff84c59000-0xffffffff84c60fff] [ 0.046340] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048557] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049025] ... version: 2 [ 0.050020] ... bit width: 48 [ 0.051017] ... generic registers: 4 [ 0.052016] ... value mask: 0000ffffffffffff [ 0.053020] ... max period: 00007fffffffffff [ 0.054008] ... fixed-purpose events: 3 [ 0.055007] ... event mask: 000000070000000f [ 0.056378] rcu: Hierarchical SRCU implementation. [ 0.059247] smp: Bringing up secondary CPUs ... [ 0.060716] x86: Booting SMP configuration: [ 0.061027] .... node #0, CPUs: #1 #2 #3 [ 0.070302] smp: Brought up 1 node, 4 CPUs [ 0.072012] smpboot: Max logical packages: 1 [ 0.073009] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.159428] node 0 deferred pages initialised in 83ms [ 0.166011] devtmpfs: initialized [ 0.167295] x86/mm: Memory block size: 128MB [ 0.172000] gcov: version magic: 0x41383552 [ 0.177495] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.187100] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.190695] pinctrl core: initialized pinctrl subsystem [ 0.198211] [ 0.198963] ************************************************************* [ 0.203013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.212023] ** ** [ 0.218023] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.224025] ** ** [ 0.227025] ** This means that this kernel is built to expose internal ** [ 0.231036] ** IOMMU data structures, which may compromise security on ** [ 0.234021] ** your system. ** [ 0.238023] ** ** [ 0.240019] ** If you see this message and you are not debugging the ** [ 0.242072] ** kernel, report this immediately to your vendor! ** [ 0.244017] ** ** [ 0.246018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.248016] ************************************************************* [ 0.252227] NET: Registered protocol family 16 [ 0.254886] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.257080] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.261093] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.263875] cpuidle: using governor menu [ 0.268323] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.273470] PCI: Using configuration type 1 for base access [ 0.279250] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.301724] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.303028] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.305078] cryptd: max_cpu_qlen set to 1000 [ 0.308689] ACPI: Added _OSI(Module Device) [ 0.310014] ACPI: Added _OSI(Processor Device) [ 0.313015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.316030] ACPI: Added _OSI(Processor Aggregator Device) [ 0.324355] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.339802] ACPI: Interpreter enabled [ 0.349061] ACPI: PM: (supports S0 S3 S4 S5) [ 0.354027] ACPI: Using IOAPIC for interrupt routing [ 0.363000] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.382325] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.425000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.432045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.447086] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.454251] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.459045] acpiphp: Slot [2] registered [ 0.462512] acpiphp: Slot [5] registered [ 0.465299] acpiphp: Slot [6] registered [ 0.466000] acpiphp: Slot [7] registered [ 0.468436] acpiphp: Slot [8] registered [ 0.470141] acpiphp: Slot [9] registered [ 0.471189] acpiphp: Slot [10] registered [ 0.473205] acpiphp: Slot [3] registered [ 0.475143] acpiphp: Slot [4] registered [ 0.477244] acpiphp: Slot [11] registered [ 0.481137] acpiphp: Slot [12] registered [ 0.482000] acpiphp: Slot [13] registered [ 0.486188] acpiphp: Slot [14] registered [ 0.490241] acpiphp: Slot [15] registered [ 0.493216] acpiphp: Slot [16] registered [ 0.494000] acpiphp: Slot [17] registered [ 0.494000] acpiphp: Slot [18] registered [ 0.494000] acpiphp: Slot [19] registered [ 0.494000] acpiphp: Slot [20] registered [ 0.494099] acpiphp: Slot [21] registered [ 0.496060] acpiphp: Slot [22] registered [ 0.497704] acpiphp: Slot [23] registered [ 0.499371] acpiphp: Slot [24] registered [ 0.503172] acpiphp: Slot [25] registered [ 0.505333] acpiphp: Slot [26] registered [ 0.506264] acpiphp: Slot [27] registered [ 0.511530] acpiphp: Slot [28] registered [ 0.516530] acpiphp: Slot [29] registered [ 0.519229] acpiphp: Slot [30] registered [ 0.524148] acpiphp: Slot [31] registered [ 0.525051] PCI host bridge to bus 0000:00 [ 0.527022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.529025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.530060] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.531058] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.532000] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.532059] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.533000] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.534000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.534000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.550020] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.554671] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.558017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.562021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.566020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.570838] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.574741] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.578047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.583010] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.590016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.607032] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.615014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.621621] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.632018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.647023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.683045] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.711822] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.736028] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.750054] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.774015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.790514] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.808018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.821031] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.847026] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.884741] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.892035] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.898100] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.947123] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.984172] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 1.005036] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.036202] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.071018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 1.091466] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.105015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.115064] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.129029] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.138193] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.140544] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.147440] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.153243] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.158242] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.168141] iommu: Default domain type: Passthrough [ 1.169373] SCSI subsystem initialized [ 1.170118] ACPI: bus type USB registered [ 1.173107] usbcore: registered new interface driver usbfs [ 1.181106] usbcore: registered new interface driver hub [ 1.182091] usbcore: registered new device driver usb [ 1.184163] pps_core: LinuxPPS API ver. 1 registered [ 1.185013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.193085] PTP clock support registered [ 1.198071] EDAC MC: Ver: 3.0.0 [ 1.200210] PCI: Using ACPI for IRQ routing [ 1.202000] NetLabel: Initializing [ 1.205016] NetLabel: domain hash size = 128 [ 1.207010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.210201] NetLabel: unlabeled traffic allowed by default [ 1.212000] vgaarb: loaded [ 1.215227] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.216011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.222388] clocksource: Switched to clocksource kvm-clock [ 1.613460] VFS: Disk quotas dquot_6.6.0 [ 1.617260] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.629126] *** VALIDATE ramfs *** [ 1.633356] *** VALIDATE hugetlbfs *** [ 1.642647] pnp: PnP ACPI init [ 1.650595] pnp: PnP ACPI: found 6 devices [ 1.731874] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.739404] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.749051] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.758545] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.769994] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.779544] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.782199] NET: Registered protocol family 2 [ 1.791942] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.815402] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.823554] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.831220] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.837462] TCP: Hash tables configured (established 65536 bind 65536) [ 1.840652] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.852986] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.863584] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.870408] NET: Registered protocol family 1 [ 1.877130] RPC: Registered named UNIX socket transport module. [ 1.882467] RPC: Registered udp transport module. [ 1.885931] RPC: Registered tcp transport module. [ 1.891153] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.896349] NET: Registered protocol family 44 [ 1.900212] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.904411] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.908242] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.910563] PCI: CLS 0 bytes, default 64 [ 1.912517] Unpacking initramfs... [ 5.827644] hrtimer: interrupt took 7372236 ns [ 5.892750] debug: unmapping init [mem 0xffff8a833cc54000-0xffff8a833ffbffff] [ 5.897040] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 5.899834] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 5.904880] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 7.864964] Initialise system trusted keyrings [ 7.868564] Key type blacklist registered [ 7.876061] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 7.909961] zbud: loaded [ 7.925507] *** VALIDATE nfs *** [ 7.926804] *** VALIDATE nfs4 *** [ 7.930939] pstore: using deflate compression [ 7.945928] Platform Keyring initialized [ 8.384925] NET: Registered protocol family 38 [ 8.389786] Key type asymmetric registered [ 8.394532] Asymmetric key parser 'x509' registered [ 8.397738] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 8.408593] io scheduler mq-deadline registered [ 8.414396] io scheduler kyber registered [ 8.418599] io scheduler bfq registered [ 8.422875] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 8.429691] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 8.439510] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 8.446933] ACPI: Power Button [PWRF] [ 8.464562] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 8.484923] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 8.519073] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 8.538906] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 8.583578] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 8.627306] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 8.687052] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 8.701883] Non-volatile memory driver v1.3 [ 8.704914] Linux agpgart interface v0.103 [ 8.801655] virtio_blk virtio1: [vda] 145872 512-byte logical blocks (74.7 MB/71.2 MiB) [ 8.807331] vda: detected capacity change from 0 to 74686464 [ 8.860670] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 8.866682] vdb: detected capacity change from 0 to 1073741824 [ 8.918365] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.926085] vdc: detected capacity change from 0 to 2621440000 [ 8.950932] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.954796] vdd: detected capacity change from 0 to 2621440000 [ 8.970373] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.973947] vde: detected capacity change from 0 to 4294967296 [ 8.989339] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.992517] vdf: detected capacity change from 0 to 4294967296 [ 8.998802] libphy: Fixed MDIO Bus: probed [ 9.008522] usbcore: registered new interface driver usbserial_generic [ 9.012715] usbserial: USB Serial support registered for generic [ 9.018054] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 9.025127] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 9.026851] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 9.029297] mousedev: PS/2 mouse device common for all mice [ 9.031718] rtc_cmos 00:05: RTC can wake from S4 [ 9.035582] rtc_cmos 00:05: registered as rtc0 [ 9.037504] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 9.040112] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 9.047418] intel_pstate: CPU model not supported [ 9.052329] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 9.057279] hid: raw HID events driver (C) Jiri Kosina [ 9.059906] usbcore: registered new interface driver usbhid [ 9.062401] usbhid: USB HID core driver [ 9.064045] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 9.068274] drop_monitor: Initializing network drop monitor service [ 9.075673] Initializing XFRM netlink socket [ 9.078084] NET: Registered protocol family 10 [ 9.081710] Segment Routing with IPv6 [ 9.083435] NET: Registered protocol family 17 [ 9.086469] mpls_gso: MPLS GSO support [ 9.105582] RAS: Correctable Errors collector initialized. [ 9.109892] AVX version of gcm_enc/dec engaged. [ 9.111904] AES CTR mode by8 optimization enabled [ 9.270745] sched_clock: Marking stable (9270718397, 0)->(11055389794, -1784671397) [ 9.279748] registered taskstats version 1 [ 9.285409] Loading compiled-in X.509 certificates [ 9.288529] zswap: loaded using pool lzo/zbud [ 9.329734] Key type big_key registered [ 9.347488] Key type encrypted registered [ 9.349956] ima: No TPM chip found, activating TPM-bypass! [ 9.355521] ima: Allocated hash algorithm: sha1 [ 9.359769] ima: No architecture policies found [ 9.363041] evm: Initialising EVM extended attributes: [ 9.365449] evm: security.selinux [ 9.367368] evm: security.ima [ 9.368897] evm: security.capability [ 9.370797] evm: HMAC attrs: 0x1 [ 9.374605] rtc_cmos 00:05: setting system clock to 2026-09-08 12:04:04 UTC (1788869044) [ 9.385250] debug: unmapping init [mem 0xffffffff85c03000-0xffffffff85dfffff] [ 9.390953] debug: unmapping init [mem 0xffffffff84982000-0xffffffff84c58fff] [ 9.397201] Write protecting the kernel read-only data: 28672k [ 9.403552] debug: unmapping init [mem 0xffffffff83003000-0xffffffff831fffff] [ 9.408294] debug: unmapping init [mem 0xffffffff83914000-0xffffffff839fffff] [ 9.503728] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 9.521759] systemd[1]: Detected virtualization kvm. [ 9.525354] systemd[1]: Detected architecture x86-64. [ 9.529041] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 9.580347] systemd[1]: No hostname configured. [ 9.585707] systemd[1]: Set hostname to . [ 9.589787] random: systemd: uninitialized urandom read (16 bytes read) [ 9.596245] systemd[1]: Initializing machine ID from random generator. [ 9.850788] random: ln: uninitialized urandom read (6 bytes read) [ 10.080901] random: systemd: uninitialized urandom read (16 bytes read) [ 10.085990] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 10.096670] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 10.137203] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 12.668024] device-mapper: uevent: version 1.0.3 [ 12.673243] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 15.171703] virtio_net virtio0 ens2: renamed from eth0 [ 15.237436] random: fast init done [ 16.259271] scsi host0: ata_piix [ 16.291459] scsi host1: ata_piix [ 16.292679] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 16.308535] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 21.907096] random: crng init done [ 21.908325] random: 7 urandom warning(s) missed due to ratelimiting [ 25.734218] dracut-initqueue[588]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached targ[ 27.978499] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) et Remote File Systems. [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 34.405815] printk: systemd: 26 output lines suppressed due to ratelimiting [ 35.595908] SELinux: Disabled at runtime. [ 35.826542] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 35.850937] systemd[1]: Detected virtualization kvm. [ 35.862977] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 38.199685] systemd[1]: initrd-switch-root.service: Succeeded. [ 38.213768] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 38.230777] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 38.239532] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 38.244545] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 38.274432] systemd[1]: Starting Journal Service... Starting Journal Service... [ 38.309592] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 38.617900] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting Huge Pages File System... [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. Starting Flush Journal to Persistent Storage... Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 40.260983] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 42.493651] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 42.655189] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 43.630261] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 43.905755] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit) [ *] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit) [ ***] A start job is running for Configur…only root support (13s / no limit) [ *** ] A start job is running for Configur…only root support (13s / no limit) [ *** ] A start job is running for Configur…only root support (14s / no limit) [*** ] A start job is running for Configur…only root support (14s / no limit)[ 53.214870] Key type dns_resolver registered [** ] A start job is running for Configur…only root support (15s / no limit) [* ] A start job is running for Configur…only root support (15s / no limit) [** ] A start job is running for Configur…only root support (16s / no limit)[ 54.334503] NFS: Registering the id_resolver key type [ 54.338422] Key type id_resolver registered [ 54.341929] Key type id_legacy registered [*** ] A start job is running for Configur…only root support (16s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg411-server login: [ 139.793213] libcfs: loading out-of-tree module taints kernel. [ 139.927657] Key type ._llcrypt registered [ 139.931204] Key type .llcrypt registered [ 140.140757] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 164.495767] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 165.991967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 166.009742] alg: No test for adler32 (adler32-zlib) [ 167.316185] Lustre: Lustre: Build Version: 2.17.55_24_g1bda257 [ 168.368595] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 170.191713] Key type lgssc registered [ 171.852198] Lustre: Echo OBD driver; http://www.lustre.org/ [ 189.407393] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 241.608980] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 257.429665] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 257.474899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 258.788818] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 258.817428] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 258.906852] Lustre: lustre-MDT0000: new disk, initializing [ 259.070826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 259.119105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 264.003649] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 278.023675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 278.133863] Lustre: 6532:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 278.158371] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 278.161885] Lustre: Skipped 1 previous similar message [ 278.286724] Lustre: lustre-MDT0001: new disk, initializing [ 278.373787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 278.409561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 278.432703] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 283.230145] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 288.410399] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 299.554878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 299.856770] Lustre: lustre-OST0000: new disk, initializing [ 299.861916] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 299.871743] Lustre: 8468:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 299.977888] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 304.674762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 304.683459] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 304.769195] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 307.722613] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 325.287039] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 325.469280] Lustre: lustre-OST0001: new disk, initializing [ 325.472585] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 325.476967] Lustre: 9543:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 325.571284] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 332.391355] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 333.989019] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 334.010137] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 334.130147] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 345.972709] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 353.776798] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 361.513155] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing check_logdir /tmp/testlogs/ [ 368.506974] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing yml_node [ 374.348431] Lustre: DEBUG MARKER: Client: 2.17.55.24 [ 378.012807] Lustre: DEBUG MARKER: MDS: 2.17.55.24 [ 380.856594] Lustre: DEBUG MARKER: OSS: 2.17.55.24 [ 382.529459] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 8 08:10:16 EDT 2026 [ 402.960641] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 404.826454] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 426.476086] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 426.487447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 426.505777] Lustre: Skipped 1 previous similar message [ 426.553791] Lustre: Skipped 3 previous similar messages [ 431.592700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 431.601717] Lustre: Skipped 3 previous similar messages [ 436.710207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 436.721116] Lustre: Skipped 3 previous similar messages [ 437.217562] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 437.460181] Lustre: server umount lustre-MDT0000 complete [ 441.994045] LustreError: 6523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869477 with bad export cookie 14558513657427266495 [ 442.005563] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 442.017160] LustreError: 6523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 446.959495] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 446.966967] LustreError: 12286:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 446.969440] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 446.990305] Lustre: Skipped 3 previous similar messages [ 447.034335] LustreError: 12286:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 448.498321] Lustre: server umount lustre-MDT0001 complete [ 453.529762] Lustre: server umount lustre-OST0000 complete [ 458.712649] Lustre: server umount lustre-OST0001 complete [ 466.599865] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 479.861614] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 524.279465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 524.671547] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 524.707481] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 524.800636] Lustre: lustre-MDT0000: new disk, initializing [ 524.925329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 524.949059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 530.463826] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 543.402348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 543.552558] Lustre: 15101:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 543.617673] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 543.620846] Lustre: Skipped 1 previous similar message [ 543.707503] Lustre: lustre-MDT0001: new disk, initializing [ 543.854477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 543.910055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 543.925824] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 549.462503] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 560.181497] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 566.561854] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 574.442717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 574.900450] Lustre: lustre-OST0000: new disk, initializing [ 574.904575] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 574.912743] Lustre: 17070:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 574.994035] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 576.340426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 576.359469] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 576.446298] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 582.703676] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 594.961355] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 600.910659] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 601.265973] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 606.355836] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 606.616712] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 612.326845] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 612.332165] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 612.342163] LustreError: Skipped 1 previous similar message [ 612.342924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 612.342930] Lustre: Skipped 1 previous similar message [ 612.362702] Lustre: Skipped 1 previous similar message [ 622.572995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 622.587074] Lustre: Skipped 4 previous similar messages [ 623.071129] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 623.340937] Lustre: server umount lustre-OST0000 complete [ 630.757118] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 630.770857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 634.498295] Lustre: server umount lustre-MDT0000 complete [ 637.920285] LustreError: 15113:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 637.954786] LustreError: 15113:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 638.751951] LustreError: 17069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869673 with bad export cookie 14558513657427268028 [ 638.758711] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 638.773289] LustreError: 17069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 639.203428] Lustre: server umount lustre-MDT0001 complete [ 645.512749] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 08:14:39 (1788869679) [ 654.026360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 654.557790] LustreError: 19339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 654.643138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 659.936506] LustreError: 19340:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 660.602456] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 664.038555] LustreError: 19339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 670.164493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 676.526560] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 685.085633] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 692.256184] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 700.560725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 708.786430] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 718.281880] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 733.679552] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 742.373880] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 742.385286] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 742.405195] Lustre: Skipped 1 previous similar message [ 742.411986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 742.421816] Lustre: Skipped 3 previous similar messages [ 748.334763] Lustre: server umount lustre-MDT0000 complete [ 750.569286] LustreError: 19340:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 750.624996] LustreError: 19340:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 752.852762] LustreError: 19321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869787 with bad export cookie 14558513657427268903 [ 752.854906] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 752.862639] LustreError: 19321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 753.363659] Lustre: server umount lustre-MDT0001 complete [ 768.234443] Lustre: server umount lustre-OST0000 complete [ 796.339204] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 810.590297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 811.092689] LustreError: 23747:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 811.171233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 811.173753] Lustre: Skipped 2 previous similar messages [ 816.884819] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 828.284796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 834.398232] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 838.684484] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 847.780117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 856.148642] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 866.352121] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 866.513453] Lustre: lustre-OST0001: new disk, initializing [ 866.516897] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 866.528257] Lustre: 26062:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 868.206981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 868.235636] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 868.369202] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 875.467436] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 886.472122] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 891.073401] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 905.696503] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 905.705284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 905.722321] Lustre: Skipped 2 previous similar messages [ 905.732343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 905.738596] Lustre: Skipped 3 previous similar messages [ 911.421496] Lustre: server umount lustre-MDT0000 complete [ 914.404516] LustreError: 26074:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 914.439851] LustreError: 26074:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 915.561237] LustreError: 26061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869950 with bad export cookie 14558513657427269757 [ 915.576554] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 916.034834] Lustre: server umount lustre-MDT0001 complete [ 931.297429] Lustre: server umount lustre-OST0000 complete [ 935.401799] Lustre: 3663:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788869954/real 1788869954] req@ffff8a83bfd38000 x1875765313670784/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788869970 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 935.441123] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 935.456761] Lustre: Skipped 3 previous similar messages [ 936.234941] Lustre: server umount lustre-OST0001 complete [ 957.397840] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 970.080753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 970.820491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 970.822539] Lustre: Skipped 3 previous similar messages [ 975.965235] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 979.948597] LustreError: 29085:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 979.976170] LustreError: 29085:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 986.831351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 993.257835] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 996.875819] Lustre: 30236:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1005.370721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1013.839160] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1023.541510] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1031.884773] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1040.634742] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1044.902976] Lustre: 32114:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1054.196139] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1054.207587] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1054.228113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1054.235951] Lustre: Skipped 4 previous similar messages [ 1058.402986] Lustre: server umount lustre-MDT0000 complete [ 1062.941106] LustreError: 29065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870098 with bad export cookie 14558513657427271045 [ 1062.949574] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1062.959446] LustreError: 29065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 1063.365308] Lustre: server umount lustre-MDT0001 complete [ 1078.536641] Lustre: server umount lustre-OST0000 complete [ 1081.381918] Lustre: 3663:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788870100/real 1788870100] req@ffff8a83841c3800 x1875765313725824/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788870116 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1081.418070] Lustre: 3663:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1082.345919] Lustre: server umount lustre-OST0001 complete [ 1091.990405] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 08:22:05 (1788870125) [ 1125.240701] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 1135.529513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1135.776412] LustreError: 34916:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1135.788370] LustreError: 34916:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 1140.598662] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1151.287863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1156.703764] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1160.419917] Lustre: 36065:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1167.607611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1175.552460] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1184.058525] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1191.076442] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1199.031292] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1203.085435] Lustre: 37943:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1209.826394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1209.834639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1209.849586] Lustre: Skipped 7 previous similar messages [ 1209.869831] Lustre: Skipped 6 previous similar messages [ 1215.655924] Lustre: server umount lustre-MDT0000 complete [ 1220.008724] LustreError: 34897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870255 with bad export cookie 14558513657427272158 [ 1220.022924] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1220.025844] LustreError: 34897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1220.670935] Lustre: server umount lustre-MDT0001 complete [ 1227.009610] Lustre: server umount lustre-OST0000 complete [ 1230.920617] Lustre: server umount lustre-OST0001 complete [ 1239.541552] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 08:24:33 (1788870273) [ 1254.569687] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 1264.490573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1264.918116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1264.927288] Lustre: Skipped 7 previous similar messages [ 1269.132699] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1277.841228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1283.457106] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1286.530179] Lustre: 41233:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1293.457112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1300.324249] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1310.089581] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1316.263301] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1324.197512] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1328.204723] Lustre: 43113:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1329.120345] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1329.125896] Lustre: Skipped 1 previous similar message [ 1334.753653] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1338.657575] Lustre: server umount lustre-MDT0000 complete [ 1342.014187] LustreError: 40063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870377 with bad export cookie 14558513657427273264 [ 1342.019445] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1342.027580] LustreError: 40063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1346.532694] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1346.545245] Lustre: Skipped 7 previous similar messages [ 1348.487675] Lustre: server umount lustre-MDT0001 complete [ 1357.169806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1367.008069] LustreError: 3660:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a82856c7b80 x1875765313828608/t0(0) o250->MGC192.168.204.111@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1372.576883] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1377.728091] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1382.602350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1387.618682] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1388.544068] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1388.554619] Lustre: Skipped 1 previous similar message [ 1396.107044] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1398.928652] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1405.948588] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1408.360975] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1419.235986] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1420.637399] Lustre: server umount lustre-MDT0000 complete [ 1424.358851] LustreError: 43987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1424.392326] LustreError: 43987:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 35 previous similar messages [ 1424.772923] LustreError: 40064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870459 with bad export cookie 14558513657427274391 [ 1424.776824] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1424.779827] LustreError: 40064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1424.976846] Lustre: server umount lustre-MDT0001 complete [ 1438.834626] Lustre: server umount lustre-OST0000 complete [ 1452.954411] Lustre: server umount lustre-OST0001 complete [ 1463.993480] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 08:28:17 (1788870497) [ 1480.728807] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 1491.370043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1496.568819] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1505.475951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1510.334888] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1513.218051] Lustre: 48871:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1520.227712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1527.188495] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1536.797641] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1543.620556] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1552.134392] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1557.077641] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1572.321718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1572.343465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1572.350162] Lustre: Skipped 24 previous similar messages [ 1576.440820] Lustre: server umount lustre-MDT0000 complete [ 1580.546456] LustreError: 47702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870615 with bad export cookie 14558513657427275182 [ 1580.558721] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1580.912587] Lustre: server umount lustre-MDT0001 complete [ 1595.469600] Lustre: server umount lustre-OST0000 complete [ 1609.702118] Lustre: server umount lustre-OST0001 complete [ 1619.791979] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 08:30:53 (1788870653) [ 1623.092869] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1625.186660] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 08:30:58 (1788870658) [ 1636.135947] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 1645.370964] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 1665.810239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1666.047969] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1666.112669] Lustre: lustre-MDT0000: new disk, initializing [ 1666.201597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1670.694711] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1680.091986] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1687.227828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1687.442838] Lustre: lustre-OST0000: new disk, initializing [ 1687.445519] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1687.449063] Lustre: Skipped 1 previous similar message [ 1687.452770] Lustre: 54950:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1688.652395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1688.658524] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1688.689145] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1693.712605] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1703.690756] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1705.654585] Lustre: DEBUG MARKER: create test files [ 1739.747548] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1739.753937] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1739.762700] Lustre: Skipped 8 previous similar messages [ 1743.718653] Lustre: server umount lustre-OST0000 complete [ 1747.094735] Lustre: server umount lustre-MDT0000 complete [ 1756.527522] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 1759.252938] Key type lgssc unregistered [ 1759.505972] LNet: 56581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1759.514612] LNetError: 56581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1759.534657] LNet: Removed LNI 192.168.204.111@tcp [ 1760.405802] Key type .llcrypt unregistered [ 1760.409626] Key type ._llcrypt unregistered [ 1787.873517] Key type ._llcrypt registered [ 1787.882136] Key type .llcrypt registered [ 1787.986466] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 1789.130173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1789.156868] alg: No test for adler32 (adler32-zlib) [ 1790.133508] Lustre: Lustre: Build Version: 2.17.55_24_g1bda257 [ 1790.345159] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 1791.991524] Key type lgssc registered [ 1792.928966] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1799.212355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1799.233332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1800.565620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1803.712553] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1809.949225] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1816.332544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1816.747962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1822.750240] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1824.950135] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1825.022272] Lustre: 59316:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 1825.037384] Lustre: 59316:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1825.045056] Lustre: 59316:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1825.056037] Lustre: 59316:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1825.070197] Lustre: 59316:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1825.079804] Lustre: 59316:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1830.130779] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1835.229676] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1840.909179] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1848.032603] Lustre: server umount lustre-OST0000 complete [ 1861.269633] Lustre: server umount lustre-MDT0000 complete [ 1872.634538] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 1875.284595] Key type lgssc unregistered [ 1875.602984] LNet: 60925:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1875.623087] LNetError: 60925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1875.634626] LNet: Removed LNI 192.168.204.111@tcp [ 1876.392565] Key type .llcrypt unregistered [ 1876.394631] Key type ._llcrypt unregistered [ 1908.166901] Key type ._llcrypt registered [ 1908.174088] Key type .llcrypt registered [ 1908.283668] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 1909.568132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1909.596439] alg: No test for adler32 (adler32-zlib) [ 1910.620169] Lustre: Lustre: Build Version: 2.17.55_24_g1bda257 [ 1910.830047] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 1912.503588] Key type lgssc registered [ 1913.610800] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1920.792490] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1920.815640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1922.240697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1926.511947] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1933.897741] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1940.420884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1940.755842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1946.926053] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1947.004713] Lustre: 63765:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1947.017766] Lustre: 63765:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1947.024854] Lustre: 63765:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1947.033287] Lustre: 63765:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1947.042624] Lustre: 63765:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1947.052469] Lustre: 63765:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1947.142986] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 1953.823588] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1958.533482] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1967.596176] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1967.596908] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1967.606411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1972.584239] Lustre: server umount lustre-OST0000 complete [ 1976.790270] Lustre: server umount lustre-MDT0000 complete [ 1987.755598] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 1990.208340] Key type lgssc unregistered [ 1990.542934] LNet: 65319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1990.553389] LNetError: 65319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1990.562634] LNet: Removed LNI 192.168.204.111@tcp [ 1991.379607] Key type .llcrypt unregistered [ 1991.382311] Key type ._llcrypt unregistered [ 2010.869596] Key type ._llcrypt registered [ 2010.872414] Key type .llcrypt registered [ 2011.029833] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 2026.819174] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 2027.681563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2027.840915] alg: No test for adler32 (adler32-zlib) [ 2028.912803] Lustre: Lustre: Build Version: 2.17.55_24_g1bda257 [ 2029.159573] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 2030.831172] Key type lgssc registered [ 2032.175154] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2068.321664] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2068.345926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2069.589393] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2069.615375] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2069.721851] Lustre: lustre-MDT0000: new disk, initializing [ 2069.810492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2069.838135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2073.883284] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2084.689224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2084.803342] Lustre: 68472:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2084.862787] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2084.874372] Lustre: Skipped 1 previous similar message [ 2084.953526] Lustre: lustre-MDT0001: new disk, initializing [ 2085.079446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2085.124191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2085.138510] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2089.118240] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2097.901317] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2104.515097] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2111.166537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2111.422082] Lustre: lustre-OST0000: new disk, initializing [ 2111.425405] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2111.430434] Lustre: 70440:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2111.519556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2116.682787] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2117.668286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2117.675391] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2117.722298] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2124.801733] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2129.385860] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2129.589933] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2133.684784] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2133.995669] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2138.094613] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2138.103927] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2138.107457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2138.116869] Lustre: Skipped 1 previous similar message [ 2143.208122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2143.213647] Lustre: Skipped 2 previous similar messages [ 2148.324996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2148.342851] Lustre: Skipped 1 previous similar message [ 2150.367722] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2150.651197] Lustre: server umount lustre-OST0000 complete [ 2157.025185] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2157.035711] LustreError: Skipped 1 previous similar message [ 2157.044709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2157.058235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2158.576932] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2163.680879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2163.685708] Lustre: Skipped 2 previous similar messages [ 2169.311504] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2169.642398] Lustre: server umount lustre-MDT0000 complete [ 2173.781895] LustreError: 68465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871208 with bad export cookie 13778972621936120096 [ 2173.786876] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2173.800893] LustreError: 68465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2174.201800] Lustre: server umount lustre-MDT0001 complete [ 2184.056621] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 08:40:17 (1788871217) [ 2194.380175] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 2235.717814] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 2247.228528] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 2288.451577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2288.712233] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2288.746675] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2288.845135] Lustre: lustre-MDT0000: new disk, initializing [ 2288.930938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2288.945456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2293.674482] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2305.906337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2306.027316] Lustre: 76465:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2306.114395] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2306.121247] Lustre: Skipped 1 previous similar message [ 2306.188635] Lustre: lustre-MDT0001: new disk, initializing [ 2306.293760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2306.342870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2306.365883] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2312.520094] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2323.422822] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2329.381423] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2337.489296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2337.799398] Lustre: lustre-OST0000: new disk, initializing [ 2337.802657] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2337.807586] Lustre: 78435:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2337.886839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2345.210160] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2347.654138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2347.667841] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2347.761984] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2356.061769] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2361.530846] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2361.734788] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2365.911977] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2366.110560] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2368.484800] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2368.520553] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2368.535160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2368.537546] Lustre: Skipped 3 previous similar messages [ 2382.305659] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2382.458542] Lustre: server umount lustre-OST0000 complete [ 2388.453525] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2388.475915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2388.486403] Lustre: Skipped 1 previous similar message [ 2388.508156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2388.517742] Lustre: Skipped 4 previous similar messages [ 2392.967117] Lustre: server umount lustre-MDT0000 complete [ 2397.125685] LustreError: 76457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871432 with bad export cookie 13778972621936120978 [ 2397.135844] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2397.142329] LustreError: 76457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2397.580741] Lustre: server umount lustre-MDT0001 complete [ 2407.930396] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 08:44:01 (1788871441) [ 2414.660950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2415.119047] LustreError: 80848:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2415.134335] LustreError: 80848:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2415.180659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2420.056273] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2420.207747] LustreError: 80849:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2425.323772] LustreError: 80848:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2429.666538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2434.782629] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2443.736107] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2451.048543] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2458.336835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2458.680734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2458.684466] Lustre: Skipped 1 previous similar message [ 2464.473715] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2473.489755] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2477.490864] Lustre: 83520:0:(genops.c:1795:obd_export_evict_by_uuid()) MGS: evicting e00e8845-4b3c-4a6f-b694-89a60d430a57 at adminstrative request [ 2480.115351] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2480.138788] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2490.341608] Lustre: MGS: Client e00e8845-4b3c-4a6f-b694-89a60d430a57 (at 0@lo) reconnecting [ 2490.351896] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0xbf38be987455d804 to 0xbf38be987455daa4 [ 2490.368810] Lustre: MGC192.168.204.111@tcp: Connection restored to 0@lo (at 0@lo) [ 2514.193876] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2514.375933] Lustre: lustre-OST0001: new disk, initializing [ 2514.381166] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2514.387737] Lustre: 83914:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2520.846323] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2523.728396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2523.739582] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2523.838516] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2532.039679] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2539.490515] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2539.499160] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2539.512580] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2539.522024] Lustre: Skipped 2 previous similar messages [ 2539.531757] Lustre: Skipped 1 previous similar message [ 2541.917816] Lustre: server umount lustre-OST0001 complete [ 2544.609928] LustreError: 82668:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2544.640466] LustreError: 82668:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2547.680260] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2547.690124] LustreError: Skipped 2 previous similar messages [ 2549.734785] LustreError: 82671:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2549.770249] LustreError: 82671:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2552.873795] Lustre: server umount lustre-OST0000 complete [ 2557.921920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2557.930414] Lustre: Skipped 2 previous similar messages [ 2562.901988] Lustre: server umount lustre-MDT0000 complete [ 2565.088560] LustreError: 81576:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2565.106248] LustreError: 81576:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2566.672187] LustreError: 83913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871601 with bad export cookie 13778972621936122532 [ 2566.679839] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2566.686454] LustreError: 83913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2566.974107] Lustre: server umount lustre-MDT0001 complete [ 2577.093461] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 08:46:50 (1788871610) [ 2578.774677] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2580.792754] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 08:46:54 (1788871614) [ 2582.241905] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2584.553401] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 08:46:57 (1788871617) [ 2586.419524] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2588.301093] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 08:47:02 (1788871622) [ 2589.945356] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2591.997563] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 08:47:05 (1788871625) [ 2593.469605] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2599.217635] loop: module loaded [ 2599.231899] loop0: detected capacity change from 0 to 4294967296 [ 2599.267715] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2600.683558] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2604.081650] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2607.714482] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2631.984631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2632.019650] LustreError: 88038:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2632.030573] LustreError: 88038:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2632.038458] LustreError: 88038:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2632.046691] LustreError: 88038:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2652.275692] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 2663.267719] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 2699.275835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2699.472444] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2699.494967] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2699.579868] Lustre: lustre-MDT0000: new disk, initializing [ 2699.654908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2699.658472] Lustre: Skipped 1 previous similar message [ 2699.668966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2704.465348] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2715.395474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2715.485927] Lustre: 90901:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2715.799340] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2720.808552] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2731.309218] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2738.100044] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2745.573693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2745.864803] Lustre: 92873:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2751.980705] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2752.071620] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2752.266443] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2761.999104] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2766.623985] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2766.838193] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2770.931684] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2771.127633] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2772.969614] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2772.983415] LustreError: Skipped 1 previous similar message [ 2772.991706] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2773.011353] Lustre: Skipped 1 previous similar message [ 2773.019556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2773.022034] Lustre: Skipped 7 previous similar messages [ 2787.295786] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2787.481907] Lustre: server umount lustre-OST0000 complete [ 2797.600326] Lustre: server umount lustre-MDT0000 complete [ 2798.564547] LustreError: 92130:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2798.596468] LustreError: 92130:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2801.709652] LustreError: 90894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871836 with bad export cookie 13778972621936123148 [ 2801.719324] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2801.719565] LustreError: 90894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2802.133699] Lustre: server umount lustre-MDT0001 complete [ 2812.217348] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 08:50:46 (1788871846) [ 2818.437752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2823.790071] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2832.625029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2833.029859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-60 [ 2833.037275] Lustre: Skipped 3 previous similar messages [ 2836.912390] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2842.810987] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2848.375804] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2854.645663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2859.502215] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2867.087539] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2872.731301] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2872.938211] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2877.586319] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2877.895774] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2885.517033] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2885.812494] Lustre: lustre-OST0001: new disk, initializing [ 2885.823382] Lustre: Skipped 2 previous similar messages [ 2885.833808] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2885.843975] Lustre: Skipped 3 previous similar messages [ 2885.856047] Lustre: 98659:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2891.921874] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2892.389417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2892.396841] Lustre: Skipped 2 previous similar messages [ 2892.399844] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2892.526219] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2901.923421] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2907.395813] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2907.633277] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2912.294231] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2912.570477] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2932.823094] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2962.718995] Lustre: Failing over lustre-MDT0000 [ 2963.102799] Lustre: server umount lustre-MDT0000 complete [ 2964.452979] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2964.453969] LustreError: 96379:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2964.460453] Lustre: Skipped 4 previous similar messages [ 2964.475254] LustreError: 96379:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 2980.833567] Lustre: 66697:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788871999/real 1788871999] req@000000001450b622 x1875767265084544/t0(0) o400->MGC192.168.204.111@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788872015 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2980.855742] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2987.842159] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2987.846215] LDISKFS-fs (dm-0): recovery complete [ 2987.862656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2990.047126] LustreError: 101475:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 2990.052183] LustreError: 66693:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d1df704d x1875767265092736/t0(0) o250->MGC192.168.204.111@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2990.092418] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xbf38be987456a518 [ 2990.109973] Lustre: MGC192.168.204.111@tcp: Connection restored to 0@lo (at 0@lo) [ 2990.306905] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2991.879888] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2991.890191] Lustre: 95287:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2993.824399] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 2995.685955] Lustre: 95290:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2995.693609] Lustre: 95290:0:(ldlm_lib.c:2084:extend_recovery_timer()) Skipped 1 previous similar message [ 2995.712935] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2995.766761] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2996.087179] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2996.412309] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2996.418814] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 1 previous similar message [ 2996.735117] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2996.745024] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 1 previous similar message [ 2996.756466] Lustre: 101508:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2996.761991] Lustre: 101508:0:(ldlm_lib.c:2084:extend_recovery_timer()) Skipped 7 previous similar messages [ 2997.720957] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2997.734093] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 3 previous similar messages [ 2998.055133] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2998.060417] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 3 previous similar messages [ 2999.017679] Lustre: 101508:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2999.033920] Lustre: 101508:0:(ldlm_lib.c:2084:extend_recovery_timer()) Skipped 6 previous similar messages [ 3000.008555] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3000.015990] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 6 previous similar messages [ 3000.345184] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3000.356176] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 6 previous similar messages [ 3003.232249] Lustre: 101508:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3003.244103] Lustre: 101508:0:(ldlm_lib.c:2084:extend_recovery_timer()) Skipped 12 previous similar messages [ 3004.206957] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3004.214306] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 12 previous similar messages [ 3004.543156] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3004.557804] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 12 previous similar messages [ 3011.520929] Lustre: 101508:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3011.532262] Lustre: 101508:0:(ldlm_lib.c:2084:extend_recovery_timer()) Skipped 25 previous similar messages [ 3012.491707] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3012.504697] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 25 previous similar messages [ 3012.825555] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3012.832202] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 25 previous similar messages [ 3027.710186] Lustre: 101508:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3027.730652] Lustre: 101508:0:(ldlm_lib.c:2084:extend_recovery_timer()) Skipped 50 previous similar messages [ 3028.700033] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3028.718609] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 50 previous similar messages [ 3029.050641] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3029.063881] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 50 previous similar messages [ 3051.255192] Lustre: 101508:0:(ldlm_lib.c:2393:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 3051.272860] Lustre: 101508:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3051.287792] Lustre: 101508:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 02875294-2ba7-4e7f-a106-30ef92b7fc90@192.168.204.11@tcp [ 3051.307740] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 3051.311986] Lustre: 101508:0:(ldlm_lib.c:1913:abort_req_replay_queue()) @@@ aborted: req@00000000e565810c x1875767249145216/t0(8589934771) o101->02875294-2ba7-4e7f-a106-30ef92b7fc90@192.168.204.11@tcp:12/0 lens 592/0 e 0 to 0 dl 1788872097 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 3051.353669] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3051.374542] Lustre: lustre-MDT0000: Denying connection for new client 02875294-2ba7-4e7f-a106-30ef92b7fc90 (at 192.168.204.11@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 3051.408207] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3051.471273] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3051.473695] Lustre: Skipped 2 previous similar messages [ 3051.478221] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 3051.557106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:78 to 0x280000401:577) [ 3051.561950] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:45 to 0x2c0000401:545) [ 3072.491020] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3072.498312] Lustre: Skipped 9 previous similar messages [ 3074.784286] Lustre: server umount lustre-OST0000 complete [ 3077.617362] LustreError: 97113:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3077.646638] LustreError: 97113:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 36 previous similar messages [ 3082.210333] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3082.219194] LustreError: Skipped 2 previous similar messages [ 3085.725815] Lustre: server umount lustre-OST0001 complete [ 3092.453655] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3092.466022] Lustre: Skipped 4 previous similar messages [ 3097.072900] Lustre: server umount lustre-MDT0000 complete [ 3101.381274] LustreError: 95272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788872136 with bad export cookie 13778972621936174360 [ 3101.391100] LustreError: 95272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3101.407716] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3102.039152] Lustre: server umount lustre-MDT0001 complete [ 3112.791932] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 08:55:46 (1788872146) [ 3120.914069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3120.922270] Lustre: *** cfs_fail_loc=197, val=0*** [ 3136.481831] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3141.601524] LustreError: 103484:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.111@tcp: failed processing log, type 4: rc = -110 [ 3146.763185] Lustre: *** cfs_fail_loc=197, val=0*** [ 3146.765267] Lustre: Skipped 65 previous similar messages [ 3167.327409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3167.333389] Lustre: Skipped 3 previous similar messages [ 3173.366174] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3180.850872] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3183.420177] Lustre: server umount lustre-OST0000 complete [ 3193.468074] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 08:57:07 (1788872227) [ 3215.021810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3215.637302] LustreError: 105105:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3215.656942] LustreError: 105105:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3220.789197] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3231.283427] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3235.428220] LustreError: 105136:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3235.437403] LustreError: 105136:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3235.442906] LustreError: 105136:0:(lod_dev.c:512:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 3235.812704] Lustre: server umount lustre-MDT0000 complete [ 3272.582524] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 3284.091887] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 3326.931274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3327.137783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3327.174557] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3327.255328] Lustre: lustre-MDT0000: new disk, initializing [ 3327.348706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3332.968082] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3345.125021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3345.306806] Lustre: 109313:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3345.617070] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3351.593689] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3362.376762] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3369.780680] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3380.255791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3380.628849] Lustre: 111281:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3388.335318] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3389.201984] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3398.832995] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3405.122788] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3405.359290] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3411.313471] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3411.764931] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3415.013014] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3415.019605] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3415.026156] LustreError: Skipped 2 previous similar messages [ 3415.036942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3415.044791] Lustre: Skipped 2 previous similar messages [ 3415.063794] Lustre: Skipped 5 previous similar messages [ 3420.668088] Lustre: server umount lustre-OST0000 complete [ 3431.988449] Lustre: server umount lustre-MDT0000 complete [ 3436.336386] LustreError: 109306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788872471 with bad export cookie 13778972621936184958 [ 3436.341359] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3436.360765] LustreError: 109306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 3436.832469] Lustre: server umount lustre-MDT0001 complete [ 3446.694317] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 09:01:20 (1788872480) [ 3456.916548] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 3460.481289] Key type lgssc unregistered [ 3460.846812] LNet: 113782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3460.864388] LNetError: 113782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3460.885685] LNet: Removed LNI 192.168.204.111@tcp [ 3462.078810] Key type .llcrypt unregistered [ 3462.081041] Key type ._llcrypt unregistered [ 3483.947113] Key type ._llcrypt registered [ 3483.951873] Key type .llcrypt registered [ 3484.215668] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 3503.564417] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 3505.547337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3505.577066] alg: No test for adler32 (adler32-zlib) [ 3506.596182] Lustre: Lustre: Build Version: 2.17.55_24_g1bda257 [ 3506.958534] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 3508.631220] Key type lgssc registered [ 3510.087608] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3562.201215] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3562.243364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3563.715834] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3563.811646] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3563.904696] Lustre: lustre-MDT0000: new disk, initializing [ 3563.974976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3564.001809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3568.695804] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3584.287868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3584.752661] Lustre: lustre-OST0000: new disk, initializing [ 3584.758288] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3584.766416] Lustre: Skipped 1 previous similar message [ 3584.790863] Lustre: 117960:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3584.904920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3593.455769] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3599.861989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3599.872279] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3599.919025] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3606.097501] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3606.316572] Lustre: lustre-OST0001: new disk, initializing [ 3606.320897] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3606.329469] Lustre: 118822:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3606.454567] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3613.828363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3613.841499] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3613.892594] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3614.521097] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3629.060209] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3629.064598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3629.083930] Lustre: Skipped 1 previous similar message [ 3629.106308] Lustre: Skipped 1 previous similar message [ 3630.901562] Lustre: server umount lustre-MDT0000 complete [ 3641.212537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3648.391932] LustreError: 117147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788872683 with bad export cookie 7817831009916101381 [ 3648.403620] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3648.414951] LustreError: 117147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3648.622157] Lustre: server umount lustre-OST0000 complete [ 3654.644997] Lustre: server umount lustre-OST0001 complete [ 3666.688783] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 09:04:59 (1788872699) [ 3696.478858] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 09:05:30 (1788872730) [ 3708.507191] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 3738.483843] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 09:06:11 (1788872771) [ 3782.851289] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 3794.262984] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 3838.674417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3839.043542] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3839.107236] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3839.212176] Lustre: lustre-MDT0000: new disk, initializing [ 3839.276440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3839.295576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3844.111143] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3856.449025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3856.628537] Lustre: 126584:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3856.711405] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3856.724617] Lustre: Skipped 1 previous similar message [ 3856.840111] Lustre: lustre-MDT0001: new disk, initializing [ 3856.941710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3856.983459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3856.999749] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3861.651110] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3871.236122] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3878.385536] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3886.286804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3886.488516] Lustre: lustre-OST0000: new disk, initializing [ 3886.492197] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3886.497193] Lustre: 128554:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3886.548558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3893.875469] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3894.880713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3894.891738] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3895.026438] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3905.120823] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3910.092885] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3910.472502] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3914.903578] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3915.091643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3920.866018] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3920.869906] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3920.881749] LustreError: Skipped 1 previous similar message [ 3920.919309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3920.928159] Lustre: Skipped 1 previous similar message [ 3923.574397] Lustre: server umount lustre-OST0000 complete [ 3931.123504] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3931.144384] Lustre: Skipped 1 previous similar message [ 3931.157163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3931.167563] Lustre: Skipped 1 previous similar message [ 3935.851950] Lustre: server umount lustre-MDT0000 complete [ 3941.345372] LustreError: 126589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3941.368549] LustreError: 126589:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3941.394032] LustreError: 128553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788872976 with bad export cookie 7817831009916102606 [ 3941.401462] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3941.425567] LustreError: 128553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3941.852372] Lustre: server umount lustre-MDT0001 complete [ 3952.619132] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 09:09:46 (1788872986) [ 3960.187255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3960.827245] LustreError: 130966:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3960.928060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3965.934759] LustreError: 130967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3966.280892] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3970.020687] LustreError: 130966:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3975.113230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3975.139611] LustreError: 130967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3980.033702] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 3988.223500] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3995.164848] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4002.451077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4002.929220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4002.936348] Lustre: Skipped 1 previous similar message [ 4009.427842] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4017.200303] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4025.234602] Lustre: *** cfs_fail_loc=159, val=0*** [ 4041.577679] Lustre: lustre-MDT0000: Client 5e6ea068-50d2-4331-ba50-84371a6619d9 (at 192.168.204.11@tcp) reconnecting [ 4043.496241] Lustre: *** cfs_fail_loc=159, val=0*** [ 4043.500993] Lustre: Skipped 5 previous similar messages [ 4058.979602] Lustre: lustre-MDT0000: Client 5e6ea068-50d2-4331-ba50-84371a6619d9 (at 192.168.204.11@tcp) reconnecting [ 4078.050537] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4078.057185] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4078.064587] LustreError: Skipped 1 previous similar message [ 4078.066391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4078.103423] Lustre: Skipped 2 previous similar messages [ 4083.181460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4083.191553] Lustre: Skipped 2 previous similar messages [ 4083.500195] Lustre: server umount lustre-OST0000 complete [ 4088.289283] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4093.435304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4093.445756] Lustre: Skipped 3 previous similar messages [ 4094.475900] Lustre: server umount lustre-MDT0000 complete [ 4098.529098] LustreError: 130961:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4098.555543] LustreError: 130961:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4099.060836] LustreError: 130948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788873134 with bad export cookie 7817831009916103488 [ 4099.069888] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4099.074582] LustreError: 130948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4099.463199] Lustre: server umount lustre-MDT0001 complete [ 4113.944526] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 4117.235510] Key type lgssc unregistered [ 4117.669237] LNet: 134748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4117.688124] LNetError: 134748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4118.758772] LNet: Removed LNI 192.168.204.111@tcp [ 4119.867129] Key type .llcrypt unregistered [ 4119.873807] Key type ._llcrypt unregistered [ 4145.933359] Key type ._llcrypt registered [ 4145.936407] Key type .llcrypt registered [ 4146.059643] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 4147.503847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4147.527180] alg: No test for adler32 (adler32-zlib) [ 4148.647208] Lustre: Lustre: Build Version: 2.17.55_24_g1bda257 [ 4148.945764] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 4150.679291] Key type lgssc registered [ 4151.766755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4159.260995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4159.279172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4160.708558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4165.396861] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4170.219878] LustreError: 136162:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4170.231964] LustreError: 136162:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4175.198279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4175.330504] LustreError: 136163:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4175.761499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4180.467426] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4189.898125] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4196.442599] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4204.111877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4204.532595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4205.565612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4210.853995] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4219.710114] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4228.844821] Lustre: *** cfs_fail_loc=159, val=0*** [ 4245.324544] Lustre: lustre-MDT0000: Client 8314ce14-642b-4ba6-be69-72f394d7a42c (at 192.168.204.11@tcp) reconnecting [ 4248.023188] Lustre: *** cfs_fail_loc=159, val=0*** [ 4264.274817] Lustre: lustre-MDT0001: Client 8314ce14-642b-4ba6-be69-72f394d7a42c (at 192.168.204.11@tcp) reconnecting [ 4266.066480] Lustre: *** cfs_fail_loc=159, val=0*** [ 4266.068290] Lustre: Skipped 3 previous similar messages [ 4282.711183] Lustre: lustre-MDT0001: Client 8314ce14-642b-4ba6-be69-72f394d7a42c (at 192.168.204.11@tcp) reconnecting [ 4288.221248] Lustre: 139197:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4291.962748] Lustre: *** cfs_fail_loc=159, val=0*** [ 4291.969450] Lustre: Skipped 4 previous similar messages [ 4307.786401] Lustre: lustre-MDT0000: Client feb78655-1045-414f-bf29-76e1fc43043e (at 192.168.204.11@tcp) reconnecting [ 4309.450111] Lustre: *** cfs_fail_loc=159, val=0*** [ 4309.455210] Lustre: Skipped 13 previous similar messages [ 4325.219874] Lustre: lustre-MDT0000: Client feb78655-1045-414f-bf29-76e1fc43043e (at 192.168.204.11@tcp) reconnecting [ 4326.844543] Lustre: 139490:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4333.044680] LustreError: lustre-OST0000-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107 [ 4333.044793] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4333.051122] LustreError: Skipped 1 previous similar message [ 4333.095375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4333.544209] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4333.545375] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4333.550581] LustreError: Skipped 1 previous similar message [ 4337.506704] Lustre: server umount lustre-OST0000 complete [ 4343.777306] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4343.778860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4343.794176] Lustre: Skipped 1 previous similar message [ 4343.808918] Lustre: Skipped 2 previous similar messages [ 4347.423436] Lustre: server umount lustre-MDT0000 complete [ 4348.909248] LustreError: 137461:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4348.920302] LustreError: 137461:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4350.967182] LustreError: 136143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788873386 with bad export cookie 10698735879341089269 [ 4350.975914] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4351.319889] Lustre: server umount lustre-MDT0001 complete [ 4363.065817] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 4365.404729] Key type lgssc unregistered [ 4365.708832] LNet: 140541:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4365.717392] LNetError: 140541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4365.746697] LNet: Removed LNI 192.168.204.111@tcp [ 4366.661259] Key type .llcrypt unregistered [ 4366.664107] Key type ._llcrypt unregistered [ 4390.862130] Key type ._llcrypt registered [ 4390.866716] Key type .llcrypt registered [ 4390.997879] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 4391.958463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4391.981525] alg: No test for adler32 (adler32-zlib) [ 4392.954418] Lustre: Lustre: Build Version: 2.17.55_24_g1bda257 [ 4393.247192] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 4394.943195] Key type lgssc registered [ 4395.908110] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4402.757236] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4402.776975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4404.153461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4409.416619] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4414.438659] LustreError: 141959:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4414.470505] LustreError: 141959:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4415.459075] LustreError: 141960:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4419.950363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4420.531313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4425.843562] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4433.980722] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4440.125095] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4446.557090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4446.861249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4449.912492] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4453.416099] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4460.977481] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4465.306994] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4471.120338] Lustre: 144544:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4474.255217] Lustre: 144594:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4475.902450] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4476.826787] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4481.631185] Lustre: 144740:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4482.522503] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4483.171290] Lustre: 144850:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4491.233490] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4491.264727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4491.275842] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4492.983973] Lustre: server umount lustre-OST0000 complete [ 4501.482324] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4501.511145] Lustre: Skipped 1 previous similar message [ 4501.514722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4501.536497] Lustre: Skipped 2 previous similar messages [ 4504.133282] Lustre: server umount lustre-MDT0000 complete [ 4506.592281] LustreError: 141954:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4506.620181] LustreError: 141954:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4508.439495] LustreError: 141939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788873543 with bad export cookie 10006736072396166092 [ 4508.446445] LustreError: 141939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4508.451391] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4508.882551] Lustre: server umount lustre-MDT0001 complete [ 4521.235416] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 4524.554261] Key type lgssc unregistered [ 4524.949486] LNet: 145906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4524.958855] LNetError: 145906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4524.979038] LNet: Removed LNI 192.168.204.111@tcp [ 4525.887208] Key type .llcrypt unregistered [ 4525.889149] Key type ._llcrypt unregistered [ 4550.511892] Key type ._llcrypt registered [ 4550.515246] Key type .llcrypt registered [ 4550.632974] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 4551.765614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4551.790916] alg: No test for adler32 (adler32-zlib) [ 4552.916589] Lustre: Lustre: Build Version: 2.17.55_24_g1bda257 [ 4553.254330] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 4554.992061] Key type lgssc registered [ 4556.103907] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4563.959561] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4563.972108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4565.386197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4569.444651] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4575.728467] LustreError: 147321:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4575.758803] LustreError: 147321:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4577.987777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4578.484164] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4582.464815] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4589.211282] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4593.906489] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4599.410523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4599.686157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4603.756664] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4605.379859] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4609.916378] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4613.209445] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4619.920397] Lustre: *** cfs_fail_loc=159, val=0*** [ 4619.931387] Lustre: Skipped 1 previous similar message [ 4636.481684] Lustre: lustre-MDT0000: Client 34082006-63c0-4354-8c08-dff13cd5d2a7 (at 192.168.204.11@tcp) reconnecting [ 4644.833637] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4644.854458] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4644.864922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4645.857336] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4645.866830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4645.872646] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4645.905413] Lustre: Skipped 1 previous similar message [ 4648.179744] Lustre: server umount lustre-OST0000 complete [ 4655.080603] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4655.089952] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4655.116882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4661.223063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4661.228122] Lustre: Skipped 3 previous similar messages [ 4666.335652] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4666.337567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4666.349746] Lustre: Skipped 1 previous similar message [ 4666.471930] Lustre: server umount lustre-MDT0000 complete [ 4669.877141] LustreError: 148061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788873705 with bad export cookie 13553706619564997144 [ 4669.879130] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4669.885166] LustreError: 148061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4670.471784] Lustre: server umount lustre-MDT0001 complete [ 4681.693380] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 4684.538738] Key type lgssc unregistered [ 4684.809112] LNet: 150983:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4684.814347] LNetError: 150983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4684.829803] LNet: Removed LNI 192.168.204.111@tcp [ 4685.753423] Key type .llcrypt unregistered [ 4685.756976] Key type ._llcrypt unregistered [ 4710.771725] Key type ._llcrypt registered [ 4710.775750] Key type .llcrypt registered [ 4710.950746] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 4712.076283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4712.210966] alg: No test for adler32 (adler32-zlib) [ 4713.305824] Lustre: Lustre: Build Version: 2.17.55_24_g1bda257 [ 4713.611976] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 4715.303896] Key type lgssc registered [ 4716.618989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4725.291696] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4725.326980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4726.868068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4731.252518] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4735.974635] LustreError: 152397:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4735.986875] LustreError: 152397:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4740.650588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4741.137762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4746.969563] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4755.162461] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4761.961799] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4769.862420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4770.481667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4771.508309] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4777.284143] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4783.966876] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4786.870269] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4807.876155] Lustre: lustre-OST0000: evicting 424de9b3-47ab-4c33-bba6-26809f13d056 (at 192.168.204.11@tcp) by administrative request [ 4807.883924] Lustre: Skipped 1 previous similar message [ 4838.378090] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4838.378334] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4838.383426] LustreError: Skipped 1 previous similar message [ 4838.385560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4838.393902] Lustre: Skipped 1 previous similar message [ 4838.412020] Lustre: Skipped 1 previous similar message [ 4843.342523] Lustre: server umount lustre-OST0000 complete [ 4848.609829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4848.613817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4848.649755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4853.729139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4853.739144] Lustre: Skipped 2 previous similar messages [ 4853.970900] Lustre: server umount lustre-MDT0000 complete [ 4858.268099] LustreError: 153138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788873893 with bad export cookie 11807212376565363696 [ 4858.291023] LustreError: 153138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4858.300700] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4858.774249] Lustre: server umount lustre-MDT0001 complete [ 4870.749640] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 4873.557221] Key type lgssc unregistered [ 4873.814456] LNet: 156464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4873.827323] LNetError: 156464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4873.851390] LNet: Removed LNI 192.168.204.111@tcp [ 4874.704966] Key type .llcrypt unregistered [ 4874.713044] Key type ._llcrypt unregistered [ 4958.409598] Key type ._llcrypt registered [ 4958.413965] Key type .llcrypt registered [ 4958.553129] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 4959.945167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4960.216964] alg: No test for adler32 (adler32-zlib) [ 4961.262783] Lustre: Lustre: Build Version: 2.17.55_24_g1bda257 [ 4961.655879] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 4963.367280] Key type lgssc registered [ 4965.144935] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4975.219834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4975.261992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4976.839843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4982.433546] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 4986.346020] LustreError: 158958:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4986.379701] LustreError: 158958:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4991.461537] LustreError: 158959:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4992.439109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4992.890406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4998.575808] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5007.194953] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5013.929377] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5021.935357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5022.370869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5029.093749] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5036.805594] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5037.755273] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 5045.955221] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 5056.495874] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5056.500215] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5056.512583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5056.516398] Lustre: Skipped 1 previous similar message [ 5061.602889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5061.608569] Lustre: Skipped 2 previous similar messages [ 5062.315404] Lustre: server umount lustre-OST0000 complete [ 5066.726103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5066.747846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5071.841108] LustreError: 158958:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5071.861683] LustreError: 158958:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5073.938471] Lustre: server umount lustre-MDT0000 complete [ 5076.972097] LustreError: 160249:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5076.992978] LustreError: 160249:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5078.130788] LustreError: 159699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788874113 with bad export cookie 17313801008470557695 [ 5078.145121] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5078.163109] LustreError: 159699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5078.690451] Lustre: server umount lustre-MDT0001 complete [ 5094.599696] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 5097.785767] Key type lgssc unregistered [ 5098.047414] LNet: 162522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5098.054960] LNetError: 162522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5098.073750] LNet: Removed LNI 192.168.204.111@tcp [ 5098.939129] Key type .llcrypt unregistered [ 5098.948397] Key type ._llcrypt unregistered [ 5128.715953] Key type ._llcrypt registered [ 5128.720599] Key type .llcrypt registered [ 5128.886691] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 5130.186160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5130.212238] alg: No test for adler32 (adler32-zlib) [ 5131.369517] Lustre: Lustre: Build Version: 2.17.55_24_g1bda257 [ 5131.739900] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 5133.359271] Key type lgssc registered [ 5134.941249] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5152.182702] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 09:29:45 (1788874185) [ 5169.125251] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 5180.633320] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 5222.395942] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5222.418881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5223.793354] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5223.867309] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5224.065718] Lustre: lustre-MDT0000: new disk, initializing [ 5224.224937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5224.241448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5229.308372] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5240.496785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5240.605225] Lustre: 166786:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5240.660294] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5240.666146] Lustre: Skipped 1 previous similar message [ 5240.778261] Lustre: lustre-MDT0001: new disk, initializing [ 5240.896348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5240.953444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5240.962027] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5246.042835] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5258.249152] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5265.213838] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5272.504276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5272.847590] Lustre: lustre-OST0000: new disk, initializing [ 5272.856178] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5272.861133] Lustre: 168758:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5272.971897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5279.360307] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5279.820549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5279.838171] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5279.963131] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5289.415397] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5300.709138] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5300.717462] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5300.727900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5300.737930] Lustre: Skipped 1 previous similar message [ 5305.827378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5305.833346] Lustre: Skipped 1 previous similar message [ 5306.107658] Lustre: server umount lustre-OST0000 complete [ 5310.945813] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5310.950269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5310.959182] Lustre: Skipped 2 previous similar messages [ 5310.971639] Lustre: Skipped 1 previous similar message [ 5316.067468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5316.081866] Lustre: Skipped 1 previous similar message [ 5321.185737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5321.194648] Lustre: Skipped 1 previous similar message [ 5324.257887] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5324.551288] Lustre: server umount lustre-MDT0000 complete [ 5328.579969] LustreError: 166778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788874363 with bad export cookie 2868468642068079245 [ 5328.582677] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5328.600626] LustreError: 166778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5329.209975] Lustre: server umount lustre-MDT0001 complete [ 5341.546689] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 5344.390528] Key type lgssc unregistered [ 5344.899186] LNet: 170665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5344.905408] LNetError: 170665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5344.919490] LNet: Removed LNI 192.168.204.111@tcp [ 5345.769912] Key type .llcrypt unregistered [ 5345.772359] Key type ._llcrypt unregistered [ 5373.288877] Key type ._llcrypt registered [ 5373.290631] Key type .llcrypt registered [ 5373.399458] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 5374.374649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5374.455276] alg: No test for adler32 (adler32-zlib) [ 5376.223325] Lustre: Lustre: Build Version: 2.17.55_24_g1bda257 [ 5376.580629] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 5378.495274] Key type lgssc registered [ 5379.949575] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5388.765976] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5388.790347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5390.203725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5395.787478] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5400.551623] LustreError: 172104:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5400.570898] LustreError: 172104:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5405.672053] LustreError: 172103:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5406.700029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5407.463259] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5413.531738] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5423.248655] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5430.102677] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5438.557834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5438.895962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5445.334561] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 5454.291324] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5460.403781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5467.113105] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5467.126905] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5467.148604] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5467.157223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.171353] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5468.281205] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5468.298517] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5468.310678] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5468.318564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5468.327923] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5469.762683] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5469.805406] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5469.852184] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5469.870105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.892077] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5471.111826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5472.343626] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5472.348932] Lustre: Skipped 1 previous similar message [ 5472.366687] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5472.377222] Lustre: Skipped 1 previous similar message [ 5472.381160] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5472.385400] Lustre: Skipped 1 previous similar message [ 5472.389529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5472.400164] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5472.409265] Lustre: Skipped 1 previous similar message [ 5473.799590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5474.978215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.972784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5477.231054] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5477.241597] Lustre: Skipped 3 previous similar messages [ 5477.268407] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5477.276402] Lustre: Skipped 3 previous similar messages [ 5477.280556] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5477.285618] Lustre: Skipped 3 previous similar messages [ 5477.289581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5477.300980] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5477.306226] Lustre: Skipped 3 previous similar messages [ 5478.462673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5479.520991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5480.485770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.436152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.463243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5483.343219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.436449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.538855] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5485.548024] Lustre: Skipped 7 previous similar messages [ 5485.569269] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5485.582764] Lustre: Skipped 7 previous similar messages [ 5485.586397] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5485.590850] Lustre: Skipped 7 previous similar messages [ 5485.594777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.608879] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5485.617792] Lustre: Skipped 7 previous similar messages [ 5486.692650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5487.808210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.825887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.862852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5490.890913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.900291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.899125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.824093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.139409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5496.181202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.244523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5498.249897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.355795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.354133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.326373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5502.258831] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5502.263975] Lustre: Skipped 15 previous similar messages [ 5502.287592] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5502.302693] Lustre: Skipped 15 previous similar messages [ 5502.308331] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5502.312762] Lustre: Skipped 15 previous similar messages [ 5502.315400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5502.328706] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5502.341468] Lustre: Skipped 15 previous similar messages [ 5503.410812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.408992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.523275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.505953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.688871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.745762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5509.921992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.934284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5511.816318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.818877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.809542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5514.903786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.799333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5516.887942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.017982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5519.246536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.322913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.616497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5522.620617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.688476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.714047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5525.632298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5526.742806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.692460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.574256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.455906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5530.547974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.536419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.607824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5533.663644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.630066] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5534.637369] Lustre: Skipped 30 previous similar messages [ 5534.654784] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5534.667127] Lustre: Skipped 30 previous similar messages [ 5534.678048] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5534.688813] Lustre: Skipped 30 previous similar messages [ 5534.692982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.702539] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5534.710126] Lustre: Skipped 30 previous similar messages [ 5535.752780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.956578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.129707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5539.074302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.176776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5541.078534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.159961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.081887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.165931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.151804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.141371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.000416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.056363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.212955] LustreError: 172123:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5549.239783] LustreError: 172123:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5549.247722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.387918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.368887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.409785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.410357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.419906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.363887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5556.423364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.301375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.246839] LustreError: 172123:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5558.254860] LustreError: 172123:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5558.260408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.088734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.178875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.097210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.903323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.781173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.797856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.595879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5565.665021] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.730198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.935348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.993577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.277833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.312438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.259185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.164893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.089185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.010979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.883980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.752936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.626053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.475775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.420442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.400182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.513828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.440828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.360048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.198532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.094398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.998929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.207704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.252571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.214972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.222669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.333738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.280061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.309560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.315021] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.334600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.345570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.318509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.300398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.240243] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5599.244891] Lustre: Skipped 64 previous similar messages [ 5599.270321] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5599.280659] Lustre: Skipped 64 previous similar messages [ 5599.285551] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5599.293479] Lustre: Skipped 64 previous similar messages [ 5599.295565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.302559] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5599.309442] Lustre: Skipped 64 previous similar messages [ 5600.106520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.982785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.842150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.695369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.748628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.605448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.341876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.032696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.863210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.462620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.117386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.871668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.613266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.267479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.970160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.948567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.947914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.689411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.475045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.293394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.406272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.848612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.054442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.773487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.655042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.378545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.380221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.293626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.466247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.593333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.003189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.077651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.183394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.376335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.199460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.375849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.409136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.141970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.104839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.178742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.148242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.289494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.467220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.346412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.449482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.237264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.072852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.731130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.416344] LustreError: 172123:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5644.423120] LustreError: 172123:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5644.432487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.098919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.800187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.520149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.302325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.018974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.771622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.589785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.600911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.540272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.733205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.773303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.895174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.873031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.854839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.819935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.712786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.530519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.458393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.243856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.078335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.864448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.903606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.799182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.712028] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.663173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.702156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.711510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.696070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.824207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.022760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.086687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.217654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.280961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.201353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.253331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.376388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.417179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.488479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.532036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.618806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.638224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.756973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.734561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.668515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.676304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.771897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.730526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.637969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.630157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.844142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.835584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.647037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.496672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.619911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.498690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.438382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.386321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.323561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.213757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.146839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.054855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.906601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.841426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.806834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.714650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.499082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.353930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.238433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.064337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.031574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5712.005459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5712.931590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.988864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.845612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.646503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.499189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.321196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.204702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.079089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.857810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.691557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.513497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.568128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.392518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.211140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.114496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.883862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.751991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.581339] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5727.589057] Lustre: Skipped 137 previous similar messages [ 5727.607144] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5727.618398] Lustre: Skipped 137 previous similar messages [ 5727.621962] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5727.626548] Lustre: Skipped 137 previous similar messages [ 5727.629198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.642581] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5727.648909] Lustre: Skipped 137 previous similar messages [ 5728.482853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.244598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.133412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.969315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.819866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.724989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.545744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.367854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.449867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.312083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.232438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.256243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.104376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5740.184143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.226398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.185299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.172395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.301101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.156129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.087734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.903568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.881748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.680558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.622639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5750.500259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.318044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.159349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.935335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.823449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.554245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.434403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.233675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.146796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.035991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.885305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.662310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.541340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.237335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.141368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.001143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.992120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.907957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.721251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.645237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.503203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.309138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.262376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.172671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.061137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.976397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.944718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.794142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.492215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.283439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.170356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.902619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.804060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.725079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.665201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.532429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.525174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.378030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.328311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.302542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.288017] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.090593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.871050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.723490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.498134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.388794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.323775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.126663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.113188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.990710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.839988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.792825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.607493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.517976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.597585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.433968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.424275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.340353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.242168] LustreError: 172123:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5801.257354] LustreError: 172123:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5801.270967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.275377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.297442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.351463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.309835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.225478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.328610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.203347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.175734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.305738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.167294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.062479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.957596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.938452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5814.975304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.863523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.854955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.833469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.759976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.742845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.746190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.788458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.744590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.610215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.498779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.644344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.434831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.313640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.194538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.068224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.819425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.620836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.296303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.185341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.957716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.818931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.629671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5835.572075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.389426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.301317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.075861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.896763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.729286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.599509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.447127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.183327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.920656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.808743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5844.595489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.449375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.444216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.379701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.119427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.809188] LustreError: 172123:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5848.820868] LustreError: 172123:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5848.827829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.598365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.542336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.392875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.431673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.255584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.185787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.015723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.913814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.630092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5857.413783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.194635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.981154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.778454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.444119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.185783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.924401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.641750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.432508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.326373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.257859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.161493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.907096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.691253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.474367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.217388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.997877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.837200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.685364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.468135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.401023] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.278683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5875.189256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.303400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.178149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.010319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.715331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.643478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.552691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5881.443831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5882.282496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.421812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.374848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.249177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.164190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.982672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.894480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.825713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.670966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.598789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.545836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.626326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.508182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.462303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.463162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.346106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.368144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.438458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.623500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.650548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5901.721868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.873474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.837569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.735915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.686368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.675189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.594868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.518917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.474550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.556587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.369424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.322417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.303538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5914.312114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.237816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.140935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.913736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.866278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.721185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5919.683775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.525097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5921.483159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.337937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5923.391260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.336913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.329386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.304800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5927.169231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5928.155867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5929.132946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5929.986712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5930.971844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.935415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.885965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5933.731362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5934.651405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5935.599899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5936.552433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5937.577654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5938.484303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.375306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.230850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5941.384397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.200347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5943.095973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5944.076749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5945.078625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.082754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5947.051282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5947.867522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.690586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.517906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5950.434820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5951.281812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5952.168058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.114920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.950418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5954.848110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5955.908217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5956.872669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5957.897376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5958.840215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5959.843279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5960.881485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5962.001432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5963.139503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5964.080508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5965.259373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5966.181237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.030044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.926264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5968.849480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5969.755805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5970.640503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5971.463496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5972.383226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5973.309442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5974.370810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.234257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.115174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5977.035554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5977.997725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.853710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.757808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5980.585690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5981.499347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5982.643200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5983.529076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.542628] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5984.550154] Lustre: Skipped 283 previous similar messages [ 5984.568097] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5984.576125] Lustre: Skipped 283 previous similar messages [ 5984.581339] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5984.586932] Lustre: Skipped 283 previous similar messages [ 5984.590994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.603405] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5984.607707] Lustre: Skipped 283 previous similar messages [ 5985.531704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.392819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5987.290057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.181948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.955554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5990.008798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.057062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5992.140345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5993.171614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5994.208416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5995.017867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5995.788568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5996.670310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5997.510714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.399537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5999.236934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.245241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6001.396045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6002.313063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6003.284120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6004.010376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6004.999419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.092970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6007.080237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6008.026229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6008.855586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6009.856275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6010.777195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6011.689771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6012.615951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6013.560154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.432845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6015.396738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6016.240799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6017.360752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6018.361985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6019.269737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6020.106845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6021.214630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6022.266355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6023.232449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6024.217420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6025.124430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6026.229950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6027.285179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6028.291955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6029.329098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6030.340356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6031.358814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6032.763731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6033.688056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.771296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6035.617937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6036.549797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6037.541318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.538919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6039.526262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6040.423510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6041.380313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6042.414225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6043.450982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6044.442428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6045.525909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6046.668842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6047.831618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6048.908888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6050.013313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6051.024603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6052.053069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6053.260756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6054.296675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6055.341344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.409420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6057.525730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6058.423591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6059.362656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6060.303909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6061.378639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6062.303994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6063.242038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.280463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6065.379131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6066.460347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6067.484244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6068.456635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6069.456790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6070.346108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6071.477772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6072.251843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6073.439508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6074.797547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.947736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6077.186979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6078.483427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6079.560870] LustreError: 172123:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6079.567918] LustreError: 172123:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6079.587467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6080.508737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6081.676153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6082.659436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6083.833109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6084.967120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6086.219737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6087.710289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6088.790038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6089.916600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6090.940153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6092.197303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6093.100404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6094.176359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6095.203798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6096.303882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6097.442463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6098.677695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6100.047550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6101.349524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6102.367629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6103.473591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6104.600566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6105.619087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6106.798533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6107.792902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6109.026670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6110.123749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6111.248606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6112.376208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6113.340292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6114.526303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6115.569620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6116.669400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6117.594888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6118.504770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6119.817197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6120.828866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6121.746821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6122.810781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6123.853069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6124.868573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6125.759776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6126.681413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6127.847887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6128.832235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6129.797025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6130.695292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6131.632052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6132.421580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6133.315305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6134.224147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6135.243824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6136.112883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6137.162379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6138.382418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6139.540783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6140.534755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6141.671317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6142.644662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6143.692522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6144.544687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6145.540057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6146.510403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6147.316816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6148.256816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6149.105735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6150.067514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6150.961044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6151.842322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6152.781324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6153.726585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6154.690604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6155.758050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6156.677294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6157.732848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6158.891288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6159.988760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6161.031952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6162.006741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6162.946894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6163.981127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6164.926194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6165.924900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6166.959862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6168.050277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6169.102752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6170.349411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6171.295641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6172.344967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6173.319652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6174.377809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6175.385976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6176.520184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6177.444723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6178.638686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6179.627316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6180.627583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6181.529834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6182.725660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6183.770987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6184.732787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6185.623060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6186.758311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6187.768491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6188.952023] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6189.880872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6190.896179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6191.935930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6192.942180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6193.866920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6194.862077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6195.826690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6196.897269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6197.861403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6198.869407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6199.865352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6200.852804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6201.637736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6202.591351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6203.464886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6204.372537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6205.231258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6206.016535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6206.906578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6207.757130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6208.708539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6209.750360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6210.735223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6211.868616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6212.946272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6213.887276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6214.873612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6215.946091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6216.995775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6217.993556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6218.894922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6219.786575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6220.604460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6221.736537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6222.885205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6224.032573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6225.072518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6226.334897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6227.283401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6228.724273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6229.996896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6230.994232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6232.005242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6233.109966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6234.047294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6235.040946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6236.051425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6237.083452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6238.167706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6239.314754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6240.565243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6241.526510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6242.742351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6243.917348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6245.009698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6245.942058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6247.411821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6248.486822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6249.776776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6251.045164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6252.299150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6253.592520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6254.594695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6255.836191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6257.044072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6258.003560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6259.025506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6259.902420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6261.056923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6262.075470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6263.321753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6264.220827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6265.446685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6266.423301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6267.427379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6268.395255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6269.502352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6270.572954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6271.772506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6272.883659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6273.873502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6274.763598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6275.756176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6276.874599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6277.970497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6279.087360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6280.190771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6281.340527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6282.592253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6283.596522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6284.872261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6286.007153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6287.081574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6288.058244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6289.062273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6290.001291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6291.316256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6292.483699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6293.711237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6294.966887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6296.086131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6297.276893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6298.697273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6299.924614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6301.003988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6302.103968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6303.230949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6304.389100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6305.568266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6306.875574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6308.244760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6309.380902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6310.348433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6311.425683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6312.401971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6313.562216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6314.836857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6316.161393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6317.315084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6318.371397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6319.775748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6320.773196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6321.455282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6322.388249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6323.364030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6324.271092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6325.157456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6326.357406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6327.403407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6328.546721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6329.606242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6330.881364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6332.172646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6333.368260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6334.682299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6335.925676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6337.189499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6338.174439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6340.009601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6341.132727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6342.402715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6343.651494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6344.754993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6346.151292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6347.418496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6348.611201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6349.721823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6350.752776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6351.584479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6352.941692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6354.173820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6355.214904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6356.535177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6357.858881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6358.947253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6360.044519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6360.838658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6361.787930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6362.780230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6364.084166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6365.384519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6366.523622] LustreError: 172123:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6366.541309] LustreError: 172123:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6366.555849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6367.485574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6368.230294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6369.223704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6370.342922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6371.115220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6372.298146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6373.215643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6374.331944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6375.151974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6376.009664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6376.913411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6377.974451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6379.099279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6380.350714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6381.356730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6382.454247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6383.454114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6384.624515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6385.526911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6386.678728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6387.664347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6388.862383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6389.856471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6390.944204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6391.921115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6392.810390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6393.641516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6394.632979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6395.563271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6396.560820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6397.617124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6398.477821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6399.479809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6400.318811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6401.258219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6402.235396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6403.395242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6404.286361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6405.340361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6406.607397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6407.592626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6408.603619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6409.527823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6410.644861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6411.729501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6412.658485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6413.752528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6414.748145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6415.712415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6416.672054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6417.826655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6418.940613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6420.032753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6421.036913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6421.995844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6423.084858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6424.132195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6425.281975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6426.250473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6427.295762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6428.325304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6429.317666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6430.418223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6431.501494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6432.430418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6433.505242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6434.423904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6435.583466] LustreError: 172123:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6435.595135] LustreError: 172123:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6435.603093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6436.583683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6437.523881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6438.427440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6439.415330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6440.293981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6441.362609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6442.174458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6443.232510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6444.271723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6445.390503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6446.355791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6447.527544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6448.541644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6449.643211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6450.519234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6451.457564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6452.179183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6453.102730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6453.998842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6903.143747] LustreError: 174595:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6903.158097] LustreError: 172123:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 7615.968947] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7615.977297] Lustre: Skipped 449 previous similar messages [ 7615.982438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7615.993831] Lustre: Skipped 1 previous similar message [ 7621.096690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7621.102551] Lustre: Skipped 1 previous similar message [ 7622.028960] Lustre: server umount lustre-OST0000 complete [ 7626.213983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7626.219117] Lustre: Skipped 1 previous similar message [ 7631.328934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7631.337914] Lustre: Skipped 1 previous similar message [ 7631.788893] Lustre: server umount lustre-MDT0000 complete [ 7635.253643] LustreError: 172085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876670 with bad export cookie 834019923978158476 [ 7635.254721] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7635.260427] LustreError: 172085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7635.515061] Lustre: server umount lustre-MDT0001 complete [ 7645.129408] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 7647.527855] Key type lgssc unregistered [ 7647.848672] LNet: 227417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7647.856808] LNetError: 227417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7647.878270] LNet: Removed LNI 192.168.204.111@tcp [ 7648.655093] Key type .llcrypt unregistered [ 7648.659523] Key type ._llcrypt unregistered [ 7672.285178] Key type ._llcrypt registered [ 7672.287279] Key type .llcrypt registered [ 7672.374313] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 7673.467483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7673.495176] alg: No test for adler32 (adler32-zlib) [ 7674.643489] Lustre: Lustre: Build Version: 2.17.55_24_g1bda257 [ 7674.948810] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 7676.631369] Key type lgssc registered [ 7677.731459] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7685.167853] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7685.189784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7686.613959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7690.919181] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 7696.872942] LustreError: 228834:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7696.884439] LustreError: 228834:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7698.456542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7698.773101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7702.652744] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 7709.986681] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7715.217602] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7721.250489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7721.648096] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7727.370711] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 7731.895299] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 7734.110566] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7739.590147] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7741.007791] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7741.023221] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 7741.033414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7741.046760] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7741.046784] LustreError: 228853:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 7744.481292] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7744.488334] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7744.501240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7746.854679] Lustre: lustre-OST0000: Not available for connect from 192.168.204.11@tcp (stopping) [ 7747.561024] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7751.945118] Lustre: lustre-OST0000: Not available for connect from 192.168.204.11@tcp (stopping) [ 7751.953485] Lustre: Skipped 2 previous similar messages [ 7757.066530] Lustre: lustre-OST0000: Not available for connect from 192.168.204.11@tcp (stopping) [ 7757.071846] Lustre: Skipped 2 previous similar messages [ 7757.280030] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7757.388969] Lustre: server umount lustre-OST0000 complete [ 7761.103147] LustreError: 228853:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 7789.997808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7790.315749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7795.326486] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 7801.525219] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7802.604551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7802.620806] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7807.787107] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7807.800798] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7814.153830] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 10 sec [ 7819.167762] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8ca793c96000.ost_server_uuid 50 [ 7820.557675] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8ca793c96000.ost_server_uuid in FULL state after 0 sec [ 7825.889253] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7825.900928] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7825.910518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7828.458647] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7830.207360] Lustre: server umount lustre-OST0000 complete [ 7837.154530] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7837.175214] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7837.202787] Lustre: Skipped 1 previous similar message [ 7837.209818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7837.219119] Lustre: Skipped 2 previous similar messages [ 7840.191288] Lustre: server umount lustre-MDT0000 complete [ 7843.811782] LustreError: 228833:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7843.836174] LustreError: 228833:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7844.118972] LustreError: 228814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876879 with bad export cookie 4083761515643364040 [ 7844.121783] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7844.132185] LustreError: 228814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7844.641779] Lustre: server umount lustre-MDT0001 complete [ 7855.541584] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 7858.227639] Key type lgssc unregistered [ 7858.528821] LNet: 234075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7858.539574] LNetError: 234075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7858.551900] LNet: Removed LNI 192.168.204.111@tcp [ 7859.388219] Key type .llcrypt unregistered [ 7859.391181] Key type ._llcrypt unregistered [ 7889.016491] Key type ._llcrypt registered [ 7889.020965] Key type .llcrypt registered [ 7889.134818] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 7890.016307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7890.041294] alg: No test for adler32 (adler32-zlib) [ 7891.075211] Lustre: Lustre: Build Version: 2.17.55_24_g1bda257 [ 7891.323463] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 7892.967800] Key type lgssc registered [ 7893.974145] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7903.168187] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7903.189589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7904.526532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7908.271873] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 7913.958139] LustreError: 235819:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7913.982821] LustreError: 235819:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7916.956435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7917.368523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7921.815847] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 7924.912288] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7931.881844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7932.307511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7938.898673] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 7947.170184] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7947.307438] Lustre: lustre-OST0001: new disk, initializing [ 7947.310083] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7947.315283] Lustre: 238130:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7947.391696] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7949.456882] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 7952.567701] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 7955.558753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7955.566241] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7955.684125] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7963.093266] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7973.551993] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7989.218915] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7989.234346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7989.260487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7991.784802] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7991.793536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7991.799061] Lustre: Skipped 2 previous similar messages [ 7991.809855] Lustre: Skipped 3 previous similar messages [ 7994.359802] Lustre: server umount lustre-MDT0000 complete [ 7996.910160] LustreError: 235819:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7996.930857] LustreError: 235819:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7997.958849] LustreError: 239127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877033 with bad export cookie 9744996138856894182 [ 7997.967982] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7997.976646] LustreError: 239127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7998.278572] Lustre: server umount lustre-MDT0001 complete [ 8012.406616] Lustre: server umount lustre-OST0000 complete [ 8026.581879] Lustre: server umount lustre-OST0001 complete [ 8048.380940] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 8058.913883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8059.373815] LustreError: 241236:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8059.446168] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8063.990780] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 8064.482154] LustreError: 241237:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8069.604221] LustreError: 241236:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8072.449996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8072.770447] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8077.007579] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 8079.843479] Lustre: 242379:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8086.554598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8092.399914] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 8097.131962] LustreError: 242733:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8100.058488] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8100.278239] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8100.282966] Lustre: Skipped 1 previous similar message [ 8104.373700] Lustre: mylustre-OST0001: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 8104.374103] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49730 to 0x280000401:49760) [ 8105.655567] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 8112.839099] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8116.211482] Lustre: 244250:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8124.908168] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8124.912217] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8124.919230] Lustre: Skipped 2 previous similar messages [ 8124.925409] Lustre: Skipped 3 previous similar messages [ 8130.021769] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8130.026451] Lustre: Skipped 3 previous similar messages [ 8130.797201] Lustre: server umount mylustre-MDT0000 complete [ 8133.886951] LustreError: 241212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877169 with bad export cookie 9744996138856896989 [ 8133.892550] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8133.899799] LustreError: 241212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8134.292933] Lustre: server umount mylustre-MDT0001 complete [ 8147.083669] Lustre: server umount mylustre-OST0000 complete [ 8160.655112] Lustre: server umount mylustre-OST0001 complete [ 8177.213059] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 8185.176594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8185.541958] LustreError: 246560:0:(ldlm_lib.c:1192:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8185.599153] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8189.245934] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 8196.557727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8200.535425] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 8203.145730] Lustre: 247704:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8208.672497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8210.044943] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49730 to 0x280000401:49792) [ 8213.812758] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 8220.644458] LustreError: 248059:0:(ldlm_lib.c:1192:target_handle_connect()) tfs-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8220.660704] LustreError: 248059:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 8221.591497] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8226.827472] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 8227.307220] Lustre: tfs-OST0001: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 8233.903936] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8237.393205] Lustre: 249573:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8247.793244] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8247.797267] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 8247.805538] Lustre: Skipped 3 previous similar messages [ 8247.811525] Lustre: Skipped 1 previous similar message [ 8249.576715] Lustre: server umount tfs-MDT0000 complete [ 8252.692676] LustreError: 246536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877287 with bad export cookie 9744996138856898879 [ 8252.700740] LustreError: 246536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8252.702131] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8252.898060] Lustre: tfs-MDT0001-lwp-OST0001: Connection to tfs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8252.905096] Lustre: Skipped 2 previous similar messages [ 8253.020566] Lustre: server umount tfs-MDT0001 complete [ 8262.609887] Lustre: server umount tfs-OST0000 complete [ 8266.192652] Lustre: server umount tfs-OST0001 complete [ 8284.432296] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 8293.158337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8293.565673] LustreError: 251820:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8293.574480] LustreError: 251820:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 8293.639135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8293.642745] Lustre: Skipped 3 previous similar messages [ 8297.313307] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 8303.989632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8307.566654] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 8310.104082] Lustre: 252961:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8315.870499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8318.265910] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49730 to 0x280000401:49824) [ 8321.406915] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 8328.726875] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8330.001763] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 8333.984901] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 8340.599399] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8343.839093] Lustre: 254831:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8350.117864] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 10:23:04 (1788877384) [ 8365.538917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8365.545836] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8365.555802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8365.560348] Lustre: Skipped 1 previous similar message [ 8379.359390] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8379.540248] Lustre: server umount lustre-MDT0000 complete [ 8382.674399] LustreError: 251796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877417 with bad export cookie 9744996138856900545 [ 8382.678153] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8382.687078] LustreError: 251796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8386.531080] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8386.532301] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8386.543080] Lustre: Skipped 4 previous similar messages [ 8386.548821] Lustre: Skipped 13 previous similar messages [ 8389.041237] Lustre: server umount lustre-MDT0001 complete [ 8397.944596] Lustre: server umount lustre-OST0000 complete [ 8406.674373] Lustre: server umount lustre-OST0001 complete [ 8411.610780] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 8419.505574] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 8446.934777] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 8446.934777] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 8446.934777] [ 8446.965230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 8447.111284] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8447.133982] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8447.203926] Lustre: lustre-MDT0000: new disk, initializing [ 8447.258522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8447.262519] Lustre: Skipped 3 previous similar messages [ 8447.276522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8450.472612] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 8458.354435] LDISKFS-fs (dm-1): Mount option "noacl" will be removed by 3.5 [ 8458.354435] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 8458.354435] [ 8458.406806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 8458.478867] Lustre: 258018:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8458.528104] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8458.530923] Lustre: Skipped 1 previous similar message [ 8458.629529] Lustre: lustre-MDT0001: new disk, initializing [ 8458.735689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8458.752089] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8462.364955] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 8470.750040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8470.951359] Lustre: lustre-OST0000: new disk, initializing [ 8470.956658] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8470.962753] Lustre: 259598:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8472.807467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8472.828973] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8472.937911] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8476.344524] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 8493.090294] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8493.514737] Lustre: lustre-OST0001: new disk, initializing [ 8493.528482] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8493.545416] Lustre: 260471:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8494.787579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8494.806926] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8494.947462] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8503.586659] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 8526.372056] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 10:25:58 (1788877558) [ 8553.002995] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 10:26:26 (1788877586) [ 8571.876220] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8571.910668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8571.920809] Lustre: Skipped 2 previous similar messages [ 8577.744640] Lustre: server umount lustre-OST0000 complete [ 8582.130462] LustreError: 259734:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8582.139967] LustreError: 259734:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 8586.720188] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8590.554961] Lustre: server umount lustre-MDT0000 complete [ 8594.379834] LustreError: 258008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877629 with bad export cookie 9744996138856902092 [ 8594.386972] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8594.392868] LustreError: 258008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8600.713493] Lustre: server umount lustre-MDT0001 complete [ 8612.857060] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 8626.307641] Lustre: server umount lustre-OST0001 complete [ 8634.077140] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 8650.188383] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 8689.999215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8690.192640] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8690.223368] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8690.348981] Lustre: lustre-MDT0000: new disk, initializing [ 8690.438369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8695.399402] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 8706.902422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8707.046976] Lustre: 265053:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8707.103567] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8707.108414] Lustre: Skipped 1 previous similar message [ 8707.189136] Lustre: lustre-MDT0001: new disk, initializing [ 8707.335579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8707.345214] Lustre: Skipped 4 previous similar messages [ 8707.416302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8707.435703] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8712.516759] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 8721.899478] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8727.987708] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8735.619833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8735.894484] Lustre: 267023:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8737.083546] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8737.243113] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8742.909791] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 8755.032627] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8767.978617] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8768.001642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8768.020879] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8768.023505] Lustre: Skipped 6 previous similar messages [ 8768.069288] Lustre: Skipped 9 previous similar messages [ 8769.397663] Lustre: server umount lustre-OST0000 complete [ 8774.115872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8774.130907] LustreError: Skipped 1 previous similar message [ 8781.067988] Lustre: server umount lustre-MDT0000 complete [ 8785.472341] LustreError: 265045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877820 with bad export cookie 9744996138856903569 [ 8785.473255] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8785.481154] LustreError: 265045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8785.963833] Lustre: server umount lustre-MDT0001 complete [ 8798.848315] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 8801.672922] Key type lgssc unregistered [ 8802.051741] LNet: 268784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8802.063054] LNetError: 268784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8803.111838] LNet: Removed LNI 192.168.204.111@tcp [ 8804.126132] Key type .llcrypt unregistered [ 8804.129974] Key type ._llcrypt unregistered [ 8833.685858] Key type ._llcrypt registered [ 8833.687737] Key type .llcrypt registered [ 8833.776694] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 8835.157084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8835.172752] alg: No test for adler32 (adler32-zlib) [ 8836.250671] Lustre: Lustre: Build Version: 2.17.55_24_g1bda257 [ 8836.582763] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 8838.383774] Key type lgssc registered [ 8839.699928] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8847.349097] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8847.361417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8848.830621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8853.330426] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 8858.082652] LustreError: 270248:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8858.092302] LustreError: 270248:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8862.168478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8862.613599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8867.904965] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 8876.160659] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8881.542157] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8888.660956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8888.955708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8893.066104] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 8896.095151] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 8905.053605] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8906.490647] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 8912.868918] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8912.872772] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8912.878445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8912.895150] Lustre: Skipped 1 previous similar message [ 8915.079643] Lustre: server umount lustre-OST0000 complete [ 8921.061639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8921.064544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8921.081778] Lustre: Skipped 1 previous similar message [ 8921.089838] Lustre: Skipped 1 previous similar message [ 8923.105238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8923.118821] Lustre: Skipped 1 previous similar message [ 8928.225413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8928.234723] Lustre: Skipped 1 previous similar message [ 8933.343211] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8933.344666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8933.360093] Lustre: Skipped 1 previous similar message [ 8933.545559] Lustre: server umount lustre-MDT0000 complete [ 8937.311878] LustreError: 270227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877972 with bad export cookie 2008170778229503395 [ 8937.317113] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8937.327393] LustreError: 270227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8937.658719] Lustre: server umount lustre-MDT0001 complete [ 8950.729523] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 8953.608438] Key type lgssc unregistered [ 8953.903087] LNet: 273834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8953.910332] LNetError: 273834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8953.927934] LNet: Removed LNI 192.168.204.111@tcp [ 8954.831389] Key type .llcrypt unregistered [ 8954.834705] Key type ._llcrypt unregistered [ 8972.029096] Key type ._llcrypt registered [ 8972.032173] Key type .llcrypt registered [ 8972.162749] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 8973.052737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8973.072908] alg: No test for adler32 (adler32-zlib) [ 8974.155952] Lustre: Lustre: Build Version: 2.17.55_24_g1bda257 [ 8974.437375] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 8976.103175] Key type lgssc registered [ 8977.375449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8986.405379] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8986.422766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9002.015448] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 9007.137296] LustreError: 275028:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.111@tcp: failed processing log, type 4: rc = -110 [ 9033.824157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9040.402366] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 9048.716883] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9052.185399] Lustre: server umount lustre-OST0000 complete [ 9076.348589] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 9079.072377] Key type lgssc unregistered [ 9079.542226] LNet: 276662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9079.551662] LNetError: 276662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9080.619926] LNet: Removed LNI 192.168.204.111@tcp [ 9081.247983] Key type .llcrypt unregistered [ 9081.251586] Key type ._llcrypt unregistered [ 9118.594110] Key type ._llcrypt registered [ 9118.601091] Key type .llcrypt registered [ 9118.822911] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 9120.198326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9120.243409] alg: No test for adler32 (adler32-zlib) [ 9121.291133] Lustre: Lustre: Build Version: 2.17.55_24_g1bda257 [ 9121.503592] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 9123.223202] Key type lgssc registered [ 9124.413938] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9137.194222] loop0: detected capacity change from 0 to 204800000 [ 9137.456502] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9137.464357] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9137.760073] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9138.802232] loop0: detected capacity change from 0 to 204800000 [ 9139.022159] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9139.039112] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9139.362645] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9140.388759] loop0: detected capacity change from 0 to 204800000 [ 9140.545826] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9140.560435] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9140.644506] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9141.629845] loop0: detected capacity change from 0 to 204800000 [ 9141.900060] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9142.970558] loop0: detected capacity change from 0 to 204800000 [ 9142.994744] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9146.428774] loop0: detected capacity change from 0 to 204800000 [ 9146.457907] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9149.874333] loop0: detected capacity change from 0 to 204800000 [ 9149.894703] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9153.432114] loop0: detected capacity change from 0 to 204800000 [ 9153.458916] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9157.410347] loop0: detected capacity change from 0 to 204800000 [ 9157.576114] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9157.615034] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9157.686912] Lustre: 280085:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9157.696634] Lustre: 280085:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 9157.705599] Lustre: 280085:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 9157.716216] Lustre: 280085:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 9157.726853] Lustre: 280085:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9157.737800] Lustre: 280085:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9157.747419] Lustre: 280085:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9157.758621] Lustre: 280085:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 9157.768299] Lustre: 280085:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 9157.779796] Lustre: 280085:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 9157.787159] Lustre: 280085:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 9157.798646] Lustre: 280085:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 9157.810220] Lustre: 280085:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 9157.838675] Lustre: 280085:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 9162.179661] Lustre: server umount MGS complete [ 9162.853486] loop0: detected capacity change from 0 to 204800000 [ 9162.898733] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9163.198301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9163.235731] Lustre: lustre-MDT0000: Aborting client recovery [ 9163.919252] loop1: detected capacity change from 0 to 204800000 [ 9164.018370] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9164.073840] Lustre: 280519:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9164.082104] Lustre: 280519:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 9164.098372] Lustre: 280519:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9164.111972] Lustre: 280519:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 9164.124179] Lustre: 280519:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9164.136273] Lustre: 280519:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 9164.148076] Lustre: 280519:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9164.180069] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9164.407632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9164.443511] Lustre: lustre-MDT0001: Aborting client recovery [ 9165.136477] loop2: detected capacity change from 0 to 204800000 [ 9165.259949] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9165.300294] Lustre: 280598:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9165.312260] Lustre: 280598:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9165.331918] Lustre: 280598:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9165.340885] Lustre: 280598:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9165.350632] Lustre: 280598:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9165.361989] Lustre: 280598:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9165.376879] Lustre: 280598:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9165.520327] Lustre: Found index 0 for lustre-OST0000, updating log [ 9165.571410] Lustre: 280616:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9165.617919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9165.631307] Lustre: lustre-OST0000: Aborting recovery [ 9166.376411] loop3: detected capacity change from 0 to 204800000 [ 9166.440716] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9166.495125] Lustre: 280681:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9166.527762] Lustre: 280681:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9166.538794] Lustre: 280681:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9166.548094] Lustre: 280681:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9166.563048] Lustre: 280681:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9166.573482] Lustre: 280681:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9166.580918] Lustre: 280681:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9166.598858] Lustre: Found index 1 for lustre-OST0001, updating log [ 9166.622342] Lustre: 280686:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9166.690685] Lustre: lustre-OST0001: Aborting recovery [ 9171.958704] LustreError: 280462:0:(lod_dev.c:512:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -108 [ 9172.964932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9172.967236] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9172.994900] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 9172.997917] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 9173.989679] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 9173.994893] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 9175.660295] LustreError: 280429:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 1233us: rc = -2 [ 9176.162276] LustreError: 280430:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 1013us: rc = -2 [ 9176.191499] LustreError: 280430:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 21 previous similar messages [ 9189.344306] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9189.345130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9189.355315] Lustre: Skipped 2 previous similar messages [ 9189.360700] Lustre: Skipped 1 previous similar message [ 9194.465604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9194.473516] Lustre: Skipped 3 previous similar messages [ 9194.764073] Lustre: server umount lustre-MDT0000 complete [ 9195.613123] LustreError: 280412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788878230 with bad export cookie 11465377560798675853 [ 9195.632447] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9199.584237] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9199.588145] LustreError: 280427:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9199.588372] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9199.597605] Lustre: Skipped 2 previous similar messages [ 9199.630713] LustreError: 280427:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9200.608806] LustreError: 280428:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9200.623498] LustreError: 280428:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9201.990383] Lustre: server umount lustre-MDT0001 complete [ 9216.991134] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9217.081562] Lustre: server umount lustre-OST0000 complete [ 9232.351118] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9232.482925] Lustre: server umount lustre-OST0001 complete [ 9239.240934] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 10:37:53 (1788878273) [ 9260.791166] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 9268.121941] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 9293.663546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9293.845364] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9293.869424] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9293.933770] Lustre: lustre-MDT0000: new disk, initializing [ 9294.019604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9294.026093] Lustre: Skipped 1 previous similar message [ 9294.040276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9297.956642] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 9306.261931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9306.367985] Lustre: 284748:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9306.437753] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9306.441929] Lustre: Skipped 1 previous similar message [ 9306.500168] Lustre: lustre-MDT0001: new disk, initializing [ 9306.583749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9306.621806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9306.634567] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9310.235432] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 9316.990695] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9321.363965] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9327.272403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9327.504900] Lustre: lustre-OST0000: new disk, initializing [ 9327.509265] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9327.519928] Lustre: 286718:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9327.604035] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9332.353153] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 9332.780075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9332.788467] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9332.880417] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9340.699410] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9345.579139] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9357.115883] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9368.784343] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9375.924451] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 9414.629967] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9414.631303] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9414.631678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9414.631687] Lustre: Skipped 3 previous similar messages [ 9414.640127] Lustre: Skipped 1 previous similar message [ 9416.460860] Lustre: server umount lustre-OST0000 complete [ 9419.234247] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9419.239683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9419.246683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9419.250022] Lustre: Skipped 1 previous similar message [ 9425.388405] Lustre: server umount lustre-MDT0000 complete [ 9427.794540] LustreError: 288463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788878462 with bad export cookie 11465377560798678562 [ 9427.796184] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9427.802080] LustreError: 288463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9428.016743] Lustre: server umount lustre-MDT0001 complete [ 9433.754153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9436.488776] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 9439.633839] Lustre: server umount MGS complete [ 9446.803869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9447.141947] LustreError: 290824:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9447.167626] LustreError: 290837:0:(qmt_pool.c:1854:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9447.221675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9449.840310] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 9452.516247] LustreError: 290825:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9455.422743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9458.327466] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 9463.049411] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9467.088710] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9471.623891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9475.574647] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 9480.784209] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9507.075461] Lustre: server umount lustre-OST0000 complete [ 9511.906269] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9511.906863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9511.907908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9511.907913] Lustre: Skipped 3 previous similar messages [ 9511.933028] Lustre: Skipped 2 previous similar messages [ 9519.112400] Lustre: server umount lustre-MDT0000 complete [ 9521.523558] LustreError: 291561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788878556 with bad export cookie 11465377560798680501 [ 9521.525657] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9521.530176] LustreError: 291561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9531.969059] Lustre: server umount lustre-MDT0001 complete [ 9538.041892] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 9539.779916] Key type lgssc unregistered [ 9540.028821] LNet: 294769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9540.033150] LNetError: 294769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9540.042176] LNet: Removed LNI 192.168.204.111@tcp [ 9540.639878] Key type .llcrypt unregistered [ 9540.642334] Key type ._llcrypt unregistered [ 9562.769821] Key type ._llcrypt registered [ 9562.771327] Key type .llcrypt registered [ 9562.843364] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 9572.995452] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 9573.672553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9573.719343] alg: No test for adler32 (adler32-zlib) [ 9574.641744] Lustre: Lustre: Build Version: 2.17.55_24_g1bda257 [ 9574.795334] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 9576.423179] Key type lgssc registered [ 9577.187206] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9597.233740] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9597.247985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9598.405470] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9598.436256] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9598.494829] Lustre: lustre-MDT0000: new disk, initializing [ 9598.570644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9598.583479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9600.839958] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 9607.241995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9607.315594] Lustre: 298703:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9607.366282] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9607.369074] Lustre: Skipped 1 previous similar message [ 9607.415292] Lustre: lustre-MDT0001: new disk, initializing [ 9607.477249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9607.505546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9607.513478] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9609.695576] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 9615.356417] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9617.981359] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9621.530561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9621.668806] Lustre: lustre-OST0000: new disk, initializing [ 9621.671564] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9621.677449] Lustre: 300667:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9621.719426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9624.406497] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 9629.238826] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9631.777647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9631.782140] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9631.840323] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9639.718641] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9643.230213] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9650.255806] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9658.214130] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 9698.274303] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9698.277751] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9698.280366] LustreError: Skipped 1 previous similar message [ 9698.281286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9698.290596] Lustre: Skipped 1 previous similar message [ 9699.776067] Lustre: server umount lustre-OST0000 complete [ 9703.393148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9703.402860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9703.406428] Lustre: Skipped 2 previous similar messages [ 9708.517047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9708.519984] Lustre: Skipped 1 previous similar message [ 9713.632748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9713.638963] Lustre: Skipped 1 previous similar message [ 9716.191104] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9716.294963] Lustre: server umount lustre-MDT0000 complete [ 9718.372234] LustreError: 302345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788878753 with bad export cookie 12138797144059160491 [ 9718.375538] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9718.378501] LustreError: 302345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9718.565110] Lustre: server umount lustre-MDT0001 complete [ 9723.797984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9725.981103] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 9728.510523] Lustre: server umount MGS complete [ 9734.646665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9734.902909] LustreError: 304503:0:(qmt_pool.c:1854:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9734.938377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9736.850877] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 9740.257371] LustreError: 304491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9740.266035] LustreError: 304491:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9741.013558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9741.201540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9743.309302] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 9747.451759] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9750.401273] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9754.189493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9754.395855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9757.537861] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 9761.674193] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9786.341433] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9786.346519] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9786.365560] Lustre: Skipped 1 previous similar message [ 9786.368696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9787.360840] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9788.998644] Lustre: server umount lustre-OST0000 complete [ 9792.480768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9792.488387] Lustre: Skipped 2 previous similar messages [ 9797.604060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9797.606300] LustreError: 304486:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9797.608257] Lustre: Skipped 4 previous similar messages [ 9797.669791] Lustre: server umount lustre-MDT0000 complete [ 9800.226257] LustreError: 304472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788878835 with bad export cookie 12138797144059162416 [ 9800.230955] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9800.242803] LustreError: 304472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9800.424963] Lustre: server umount lustre-MDT0001 complete [ 9807.078457] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [ 9808.611620] Key type lgssc unregistered [ 9808.809813] LNet: 308432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9808.814881] LNetError: 308432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9808.828772] LNet: Removed LNI 192.168.204.111@tcp [ 9809.320232] Key type .llcrypt unregistered [ 9809.322550] Key type ._llcrypt unregistered [ 9822.459429] Key type ._llcrypt registered [ 9822.461389] Key type .llcrypt registered [ 9822.551367] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [ 9831.691835] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 9832.131937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9832.176125] alg: No test for adler32 (adler32-zlib) [ 9833.100407] Lustre: Lustre: Build Version: 2.17.55_24_g1bda257 [ 9833.254296] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [ 9834.871275] Key type lgssc registered [ 9835.452771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9849.683766] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [ 9858.795832] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9858.816338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9859.950155] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9859.971313] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9860.027545] Lustre: lustre-MDT0000: new disk, initializing [ 9860.067211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9860.078666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9862.443990] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 9869.186424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9869.358492] Lustre: lustre-OST0000: new disk, initializing [ 9869.361669] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9869.364740] Lustre: Skipped 1 previous similar message [ 9869.368543] Lustre: 312456:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9869.414520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9872.200359] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [ 9873.297601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9873.302325] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9873.319750] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10367.550611] Lustre: server umount lustre-MDT0000 complete [10369.479726] LustreError: 311645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788879404 with bad export cookie 2976313051919227332 [10369.488890] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10379.849256] Lustre: server umount lustre-OST0000 complete [10391.696274] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [10396.307930] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [10414.010580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10414.151710] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10414.177990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10414.224656] Lustre: lustre-MDT0000: new disk, initializing [10414.260901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10414.273653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10416.160764] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [10421.821745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10421.881800] Lustre: 316728:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10421.915856] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10421.920192] Lustre: Skipped 1 previous similar message [10421.964364] Lustre: lustre-MDT0001: new disk, initializing [10422.026825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10422.046162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10422.052532] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10423.963992] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [10428.744219] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10431.488067] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10435.037039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10435.237385] Lustre: lustre-OST0000: new disk, initializing [10435.240371] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10435.244834] Lustre: 318696:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10435.300754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10438.142253] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [10440.706853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10440.711437] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10440.756299] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10442.822728] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10444.786232] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10444.911867] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10447.071191] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10447.189579] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10450.915636] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10450.917687] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10450.919296] LustreError: Skipped 1 previous similar message [10450.920850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10450.926033] Lustre: Skipped 1 previous similar message [10450.935398] Lustre: Skipped 1 previous similar message [10454.268470] Lustre: server umount lustre-OST0000 complete [10458.080442] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10458.084556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10458.091970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10461.153997] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10461.154053] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10466.272362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10466.275852] Lustre: Skipped 2 previous similar messages [10470.367890] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10470.462858] Lustre: server umount lustre-MDT0000 complete [10472.386784] LustreError: 316719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788879507 with bad export cookie 2976313051922168123 [10472.390317] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10472.395430] LustreError: 316719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10472.551683] Lustre: server umount lustre-MDT0001 complete [10474.209610] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [10475.002951] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 10:58:29 (1788879509) [10478.325920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10478.629616] LustreError: 321009:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10478.640646] LustreError: 321009:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10478.681733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10480.654547] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [10483.680686] LustreError: 321010:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10484.556686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10486.641494] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [10490.054145] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10492.771142] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10495.963824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10496.137143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10496.141668] Lustre: Skipped 1 previous similar message [10498.810808] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [10502.010160] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10505.341924] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10505.425318] Lustre: lustre-OST0001: new disk, initializing [10505.428634] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10505.433900] Lustre: 323842:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10508.219721] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [10513.044192] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10513.153511] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10515.472375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10515.478297] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10515.505751] LustreError: 324838:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10515.512077] LustreError: 324838:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [10515.641245] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8ca787523800.ost_server_uuid 50 [10516.299243] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8ca787523800.ost_server_uuid in FULL state after 0 sec [10516.513705] LustreError: 324844:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10517.538240] LustreError: 324844:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10517.542526] LustreError: 324844:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [10518.662831] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8ca787523800.ost_server_uuid 50 [10519.367337] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8ca787523800.ost_server_uuid in FULL state after 0 sec [10519.586445] LustreError: 324844:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10519.591280] LustreError: 324844:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [10520.624429] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10535.911470] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10535.913339] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10535.916695] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10535.922731] Lustre: Skipped 1 previous similar message [10546.143394] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10546.148359] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10546.155575] Lustre: Skipped 5 previous similar messages [10546.200789] Lustre: server umount lustre-OST0001 complete [10546.950076] LustreError: 322828:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10549.729027] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10549.732956] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10549.738710] Lustre: Skipped 1 previous similar message [10551.269122] LustreError: 322827:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10551.276718] LustreError: 322827:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10554.864267] Lustre: server umount lustre-OST0000 complete [10561.506202] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10561.508568] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10561.513033] LustreError: Skipped 1 previous similar message [10561.519576] Lustre: Skipped 2 previous similar messages [10562.814807] Lustre: server umount lustre-MDT0000 complete [10564.626733] LustreError: 320991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788879599 with bad export cookie 2976313051922169005 [10564.630976] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10564.633991] LustreError: 320991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10564.795510] Lustre: server umount lustre-MDT0001 complete [10569.454922] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [10570.790376] Key type lgssc unregistered [10570.944767] LNet: 326527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10570.949192] LNetError: 326527:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10570.956459] LNet: Removed LNI 192.168.204.111@tcp [10571.365379] Key type .llcrypt unregistered [10571.366769] Key type ._llcrypt unregistered [10583.102408] Key type ._llcrypt registered [10583.104402] Key type .llcrypt registered [10583.156424] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [10583.662916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10583.678666] alg: No test for adler32 (adler32-zlib) [10584.605145] Lustre: Lustre: Build Version: 2.17.55_24_g1bda257 [10584.748348] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [10586.352448] Key type lgssc registered [10586.871853] Lustre: Echo OBD driver; http://www.lustre.org/ [10590.742126] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10590.753421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10592.007482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10593.986971] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [10597.583468] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10600.852032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10601.065513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10603.053970] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [10607.797371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10607.969973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10610.590683] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [10611.047947] LustreError: 329602:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10611.058591] LustreError: 329602:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10613.750037] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10616.291356] LustreError: 329604:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10616.735987] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10616.845414] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10617.902380] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [10617.910075] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [10619.469653] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [10622.999210] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10626.365637] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [10626.493817] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10709.988177] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10709.988375] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10709.992599] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10709.994781] LustreError: Skipped 1 previous similar message [10709.998494] Lustre: Skipped 1 previous similar message [10715.105587] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10715.109635] Lustre: Skipped 1 previous similar message [10715.588284] Lustre: server umount lustre-OST0001 complete [10718.688642] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10718.692908] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10718.697254] Lustre: Skipped 1 previous similar message [10718.699707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10720.225587] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10720.228126] LustreError: 329604:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10720.229428] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10720.237218] LustreError: 329604:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10723.808652] Lustre: server umount lustre-OST0000 complete [10728.928262] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10728.934173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10728.940684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10728.943030] Lustre: Skipped 2 previous similar messages [10731.615974] Lustre: server umount lustre-MDT0000 complete [10733.038373] LustreError: 328855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788879768 with bad export cookie 11560805628009921401 [10733.040659] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10733.042983] LustreError: 328855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10733.191193] Lustre: server umount lustre-MDT0001 complete [10737.361192] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [10738.501808] Key type lgssc unregistered [10738.647433] LNet: 332879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10738.650657] LNetError: 332879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10738.665336] LNet: Removed LNI 192.168.204.111@tcp [10739.040432] Key type .llcrypt unregistered [10739.041356] Key type ._llcrypt unregistered [10759.421190] Key type ._llcrypt registered [10759.422578] Key type .llcrypt registered [10759.489323] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [10760.113330] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10760.133215] alg: No test for adler32 (adler32-zlib) [10761.043239] Lustre: Lustre: Build Version: 2.17.55_24_g1bda257 [10761.162552] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [10762.767399] Key type lgssc registered [10763.335263] Lustre: Echo OBD driver; http://www.lustre.org/ [10767.898588] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10767.908070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10769.184661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10771.072334] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [10775.154573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10775.362475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10777.196445] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [10778.556926] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10781.596907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10781.792942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10784.339125] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [10785.894842] LustreError: 336628:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10785.902725] LustreError: 336628:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10788.149110] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10788.259494] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10790.897619] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [10794.478702] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [10794.485854] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [10800.053425] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10802.066391] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10806.239805] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10806.242687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10806.247270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10809.826159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10809.829770] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10809.832995] Lustre: Skipped 1 previous similar message [10812.388224] Lustre: server umount lustre-MDT0000 complete [10813.949311] LustreError: 336652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788879849 with bad export cookie 1076044674639465417 [10813.951567] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10813.952579] LustreError: 336652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10814.118071] Lustre: server umount lustre-MDT0001 complete [10826.388994] Lustre: server umount lustre-OST0000 complete [10838.184124] Lustre: server umount lustre-OST0001 complete [10845.797596] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [10850.084494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10850.258516] LustreError: 340405:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10850.285914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10851.978684] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [10855.392252] LustreError: 340406:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10856.837817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10859.568717] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [10861.159555] Lustre: 341544:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10864.555324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10864.762084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10864.765084] Lustre: Skipped 1 previous similar message [10867.364894] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [10867.815229] LustreError: 341899:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10871.063574] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10873.728468] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [10875.312742] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [10875.313263] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [10882.916940] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10884.385526] Lustre: 343415:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10887.648552] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10887.652540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10887.658793] Lustre: Skipped 1 previous similar message [10887.661397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10887.664161] Lustre: Skipped 3 previous similar messages [10890.720529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10890.722199] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10890.729676] Lustre: Skipped 2 previous similar messages [10891.815532] Lustre: server umount lustre-MDT0000 complete [10893.291649] LustreError: 340386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788879928 with bad export cookie 1076044674639467104 [10893.295533] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10893.296624] LustreError: 340386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10893.439584] Lustre: server umount lustre-MDT0001 complete [10904.013403] Lustre: server umount lustre-OST0000 complete [10915.017592] Lustre: server umount lustre-OST0001 complete [10916.749244] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [10917.495716] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [10918.300122] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 11:05:52 (1788879952) [10928.455773] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [10932.390439] loop0: detected capacity change from 0 to 1099511627776 [10932.585094] SGI XFS with ACLs, security attributes, quota, no debug enabled [10932.589450] *** VALIDATE xfs *** [10932.595262] XFS (loop0): Mounting V5 Filesystem [10932.607226] XFS (loop0): Ending clean mount [10933.214578] loop1: detected capacity change from 0 to 18691697672192 [10964.759864] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_hostid [10968.509244] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [10982.788592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10982.894062] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10982.908778] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10982.952610] Lustre: lustre-MDT0000: new disk, initializing [10982.983196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10982.986319] Lustre: Skipped 1 previous similar message [10982.993745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10984.525547] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [10989.267462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10989.314547] Lustre: 348527:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10989.340251] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10989.342390] Lustre: Skipped 1 previous similar message [10989.386341] Lustre: lustre-MDT0001: new disk, initializing [10989.438778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10989.442872] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10990.995089] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [10995.303195] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10997.420420] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11000.027424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11000.173238] Lustre: lustre-OST0000: new disk, initializing [11000.175694] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11000.179455] Lustre: 350495:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11002.181790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11002.186735] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11002.226714] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11002.640813] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [11007.041364] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11008.794561] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11008.887512] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11010.566802] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11010.653118] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11012.578668] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11012.580920] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11012.581981] LustreError: Skipped 1 previous similar message [11012.589186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11012.593128] Lustre: Skipped 4 previous similar messages [11017.501778] Lustre: server umount lustre-OST0000 complete [11020.256796] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11022.817120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11022.818364] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11022.820696] Lustre: Skipped 1 previous similar message [11022.828738] Lustre: Skipped 2 previous similar messages [11025.202465] Lustre: server umount lustre-MDT0000 complete [11026.898092] LustreError: 348518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788880062 with bad export cookie 1076044674639468637 [11026.900193] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11026.903827] LustreError: 348518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11027.075164] Lustre: server umount lustre-MDT0001 complete [11030.791918] XFS (loop0): Unmounting Filesystem [11032.758111] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 11:07:47 (1788880067) [11035.302221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11035.503713] LustreError: 353051:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11035.537489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11035.540224] Lustre: Skipped 2 previous similar messages [11037.169720] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [11040.915948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11042.847741] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [11045.805581] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11048.270639] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11051.040812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11053.741150] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [11057.148275] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11067.872780] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11067.877610] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11067.885581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11072.423083] Lustre: server umount lustre-OST0000 complete [11088.352068] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11088.427289] Lustre: server umount lustre-MDT0000 complete [11090.278705] LustreError: 353032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788880125 with bad export cookie 1076044674639469519 [11090.281505] LustreError: MGC192.168.204.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11090.283662] LustreError: 353032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11090.466039] Lustre: server umount lustre-MDT0001 complete [11095.061929] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing unload_modules_local [11096.352274] Key type lgssc unregistered [11096.507547] LNet: 356677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11096.510847] LNetError: 356677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11096.521468] LNet: Removed LNI 192.168.204.111@tcp [11096.899163] Key type .llcrypt unregistered [11096.900663] Key type ._llcrypt unregistered [11109.600243] Key type ._llcrypt registered [11109.603427] Key type .llcrypt registered [11109.655948] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing load_modules_local [11110.021501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11110.045861] alg: No test for adler32 (adler32-zlib) [11110.986531] Lustre: Lustre: Build Version: 2.17.55_24_g1bda257 [11111.115081] LNet: Added LNI 192.168.204.111@tcp [8/256/0/180] [11112.727237] Key type lgssc registered [11113.247698] Lustre: Echo OBD driver; http://www.lustre.org/ [11116.635974] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11116.648174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11117.878535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11119.532584] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [11123.141722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11123.301679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11124.953832] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [11127.664172] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11129.759254] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11132.237527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11132.377857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11134.539941] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [11137.235679] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11152.653465] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.11@tcp (stopping) [11152.864829] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11152.869981] Lustre: Skipped 1 previous similar message [11156.447141] Lustre: server umount lustre-MDT0001 complete [11157.766347] LustreError: 358245:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11157.773600] LustreError: 358245:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11160.402703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11160.438575] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11160.441147] Lustre: Found index 1 for lustre-MDT0001, updating log [11160.454769] Lustre: 358209:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11160.570628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11162.052700] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [11165.669291] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11166.693579] LustreError: 361950:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [11166.697088] LustreError: 361950:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [11166.700761] LustreError: 358212:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [11166.718719] LustreError: 361953:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.111@tcp: cfg command failed: rc = -17 [11166.722152] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [11166.722152] [11467.745205] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11467.746517] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11467.752349] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11467.756445] Lustre: Skipped 3 previous similar messages [11470.093524] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.11@tcp (stopping) [11471.195192] Lustre: server umount lustre-MDT0001 complete [11472.866457] LustreError: 358948:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11472.875791] LustreError: 358948:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11475.205447] LustreError: 358215:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11475.507634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11475.551389] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11475.553385] Lustre: Found index 1 for lustre-MDT0001, updating log [11475.569054] Lustre: 358207:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11475.701442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11477.381329] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [11481.072058] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11484.646855] LustreError: 374373:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [11484.650869] LustreError: 374373:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [11484.654678] LustreError: 358212:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [11484.657076] LustreError: 358212:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [11484.676152] LustreError: 374376:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [11484.679068] LustreError: 374376:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [11484.681586] LustreError: 374376:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.111@tcp: cfg command failed: rc = -17 [11484.684739] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.204.111@tcp [11484.684739] [11782.411713] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.11@tcp (stopping) [11783.137310] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11783.137614] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11783.146951] Lustre: Skipped 1 previous similar message [11787.524832] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.11@tcp (stopping) [11787.527649] Lustre: Skipped 2 previous similar messages [11794.399173] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11794.456916] Lustre: server umount lustre-MDT0001 complete [11797.764127] LustreError: 365130:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.11@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11797.930687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11797.964794] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11797.966546] Lustre: Found index 1 for lustre-MDT0001, updating log [11797.978032] Lustre: 358207:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11798.096345] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [11799.422671] Lustre: DEBUG MARKER: oleg411-server.virtnet: executing set_default_debug -1 all [11803.112728] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11807.718108] LustreError: 386746:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [11807.722070] LustreError: 386746:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [11807.725653] LustreError: 358212:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [11807.727800] LustreError: 358212:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [11807.746157] LustreError: 386749:0:(lod_lov.c:210:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [11807.752318] LustreError: 386749:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.111@tcp: cfg command failed: rc = -114 [11807.755804] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [11807.755804]