[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 466606152 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003401] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008487] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.009018] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.010011] pid_max: default: 32768 minimum: 301 [ 0.011116] LSM: Security Framework initializing [ 0.012063] Yama: becoming mindful. [ 0.013029] SELinux: Initializing. [ 0.014063] *** VALIDATE selinux *** [ 0.022299] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027142] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028164] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029118] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030112] *** VALIDATE tmpfs *** [ 0.032467] *** VALIDATE proc *** [ 0.033235] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.035284] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036165] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038030] Spectre V2 : User space: Vulnerable [ 0.039010] Speculative Store Bypass: Vulnerable [ 0.042167] debug: unmapping init [mem 0xffffffffa0a59000-0xffffffffa0a60fff] [ 0.045285] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046554] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047023] ... version: 2 [ 0.048015] ... bit width: 48 [ 0.049013] ... generic registers: 4 [ 0.050014] ... value mask: 0000ffffffffffff [ 0.051015] ... max period: 00007fffffffffff [ 0.052016] ... fixed-purpose events: 3 [ 0.053012] ... event mask: 000000070000000f [ 0.054366] rcu: Hierarchical SRCU implementation. [ 0.056551] smp: Bringing up secondary CPUs ... [ 0.057600] x86: Booting SMP configuration: [ 0.058028] .... node #0, CPUs: #1 #2 #3 [ 0.061424] smp: Brought up 1 node, 4 CPUs [ 0.063009] smpboot: Max logical packages: 1 [ 0.064018] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.238071] node 0 deferred pages initialised in 172ms [ 0.242012] devtmpfs: initialized [ 0.243269] x86/mm: Memory block size: 128MB [ 0.245847] gcov: version magic: 0x41383552 [ 0.246624] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.251151] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.254425] pinctrl core: initialized pinctrl subsystem [ 0.256358] [ 0.256984] ************************************************************* [ 0.260021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.262018] ** ** [ 0.265023] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.268024] ** ** [ 0.270018] ** This means that this kernel is built to expose internal ** [ 0.273021] ** IOMMU data structures, which may compromise security on ** [ 0.276024] ** your system. ** [ 0.278018] ** ** [ 0.281021] ** If you see this message and you are not debugging the ** [ 0.283030] ** kernel, report this immediately to your vendor! ** [ 0.286022] ** ** [ 0.288020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.291023] ************************************************************* [ 0.294760] NET: Registered protocol family 16 [ 0.297557] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.300088] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.303095] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.307224] cpuidle: using governor menu [ 0.308696] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.311350] PCI: Using configuration type 1 for base access [ 0.314122] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.325117] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.328049] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.331210] cryptd: max_cpu_qlen set to 1000 [ 0.335259] ACPI: Added _OSI(Module Device) [ 0.336028] ACPI: Added _OSI(Processor Device) [ 0.338018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.339011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.344000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.351292] ACPI: Interpreter enabled [ 0.352055] ACPI: PM: (supports S0 S3 S4 S5) [ 0.353013] ACPI: Using IOAPIC for interrupt routing [ 0.355117] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.359409] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.369182] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.371034] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.374021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.377082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.383278] acpiphp: Slot [2] registered [ 0.385249] acpiphp: Slot [5] registered [ 0.387110] acpiphp: Slot [6] registered [ 0.389116] acpiphp: Slot [7] registered [ 0.390124] acpiphp: Slot [8] registered [ 0.391112] acpiphp: Slot [9] registered [ 0.393135] acpiphp: Slot [10] registered [ 0.395116] acpiphp: Slot [3] registered [ 0.396107] acpiphp: Slot [4] registered [ 0.397092] acpiphp: Slot [11] registered [ 0.399110] acpiphp: Slot [12] registered [ 0.400093] acpiphp: Slot [13] registered [ 0.402131] acpiphp: Slot [14] registered [ 0.404157] acpiphp: Slot [15] registered [ 0.405124] acpiphp: Slot [16] registered [ 0.407165] acpiphp: Slot [17] registered [ 0.409161] acpiphp: Slot [18] registered [ 0.410168] acpiphp: Slot [19] registered [ 0.412149] acpiphp: Slot [20] registered [ 0.414130] acpiphp: Slot [21] registered [ 0.415139] acpiphp: Slot [22] registered [ 0.417131] acpiphp: Slot [23] registered [ 0.418144] acpiphp: Slot [24] registered [ 0.420131] acpiphp: Slot [25] registered [ 0.422172] acpiphp: Slot [26] registered [ 0.423130] acpiphp: Slot [27] registered [ 0.425141] acpiphp: Slot [28] registered [ 0.426108] acpiphp: Slot [29] registered [ 0.428124] acpiphp: Slot [30] registered [ 0.429104] acpiphp: Slot [31] registered [ 0.430133] PCI host bridge to bus 0000:00 [ 0.432033] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.435039] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.437035] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.440038] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.444039] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.447039] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.450175] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.452837] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.456128] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.467813] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.475754] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.478019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.480027] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.484016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.486682] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.489841] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.492058] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.495894] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.501024] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.513875] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.520771] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.525380] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.532016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.540021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.556015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.565889] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.576021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.585021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.603016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.614039] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.621016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.628016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.650988] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.664017] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.675026] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.682022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.700026] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.711489] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.721016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.732018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.746017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.759692] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.766024] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.773020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.797023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.812411] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.815379] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.817873] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.820477] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.823341] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.829036] iommu: Default domain type: Passthrough [ 0.830555] SCSI subsystem initialized [ 0.832316] ACPI: bus type USB registered [ 0.836140] usbcore: registered new interface driver usbfs [ 0.838171] usbcore: registered new interface driver hub [ 0.840162] usbcore: registered new device driver usb [ 0.842243] pps_core: LinuxPPS API ver. 1 registered [ 0.845026] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.849094] PTP clock support registered [ 0.852069] EDAC MC: Ver: 3.0.0 [ 0.854073] PCI: Using ACPI for IRQ routing [ 0.855712] NetLabel: Initializing [ 0.857016] NetLabel: domain hash size = 128 [ 0.859013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.861105] NetLabel: unlabeled traffic allowed by default [ 0.864156] vgaarb: loaded [ 0.865346] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.867019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.873833] clocksource: Switched to clocksource kvm-clock [ 0.977514] VFS: Disk quotas dquot_6.6.0 [ 0.980532] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.983229] *** VALIDATE ramfs *** [ 0.984563] *** VALIDATE hugetlbfs *** [ 0.986336] pnp: PnP ACPI init [ 0.988714] pnp: PnP ACPI: found 6 devices [ 1.003903] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.008358] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.010925] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.013374] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.015599] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.018095] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.020905] NET: Registered protocol family 2 [ 1.023296] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.028185] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.030954] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.035681] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.038768] TCP: Hash tables configured (established 65536 bind 65536) [ 1.042196] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.046231] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.049515] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.053318] NET: Registered protocol family 1 [ 1.057043] RPC: Registered named UNIX socket transport module. [ 1.059405] RPC: Registered udp transport module. [ 1.061171] RPC: Registered tcp transport module. [ 1.063390] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.066810] NET: Registered protocol family 44 [ 1.068572] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.071037] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.073030] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.075433] PCI: CLS 0 bytes, default 64 [ 1.078424] Unpacking initramfs... [ 2.499676] debug: unmapping init [mem 0xffff8d0dbcc54000-0xffff8d0dbffbffff] [ 2.507922] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.510565] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.515655] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 3.017952] Initialise system trusted keyrings [ 3.019611] Key type blacklist registered [ 3.022080] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.032401] zbud: loaded [ 3.035877] *** VALIDATE nfs *** [ 3.037391] *** VALIDATE nfs4 *** [ 3.039386] pstore: using deflate compression [ 3.043820] Platform Keyring initialized [ 3.158857] NET: Registered protocol family 38 [ 3.161265] Key type asymmetric registered [ 3.163310] Asymmetric key parser 'x509' registered [ 3.165396] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.168677] io scheduler mq-deadline registered [ 3.170725] io scheduler kyber registered [ 3.172899] io scheduler bfq registered [ 3.175358] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.178724] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.182209] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.185882] ACPI: Power Button [PWRF] [ 3.190751] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.198219] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.210148] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.217124] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.231531] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.262480] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.292854] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.298700] Non-volatile memory driver v1.3 [ 3.300205] Linux agpgart interface v0.103 [ 3.331986] virtio_blk virtio1: [vda] 150688 512-byte logical blocks (77.2 MB/73.6 MiB) [ 3.334931] vda: detected capacity change from 0 to 77152256 [ 3.351228] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.354646] vdb: detected capacity change from 0 to 1073741824 [ 3.367742] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.371401] vdc: detected capacity change from 0 to 2621440000 [ 3.385771] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.388166] vdd: detected capacity change from 0 to 2621440000 [ 3.405499] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.408653] vde: detected capacity change from 0 to 4294967296 [ 3.424883] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.427489] vdf: detected capacity change from 0 to 4294967296 [ 3.433746] libphy: Fixed MDIO Bus: probed [ 3.444394] usbcore: registered new interface driver usbserial_generic [ 3.447281] usbserial: USB Serial support registered for generic [ 3.450061] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.453525] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.455053] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.458047] mousedev: PS/2 mouse device common for all mice [ 3.461690] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.465375] rtc_cmos 00:05: RTC can wake from S4 [ 3.468710] rtc_cmos 00:05: registered as rtc0 [ 3.468772] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.470406] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.476446] intel_pstate: CPU model not supported [ 3.477110] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.483235] hid: raw HID events driver (C) Jiri Kosina [ 3.485230] usbcore: registered new interface driver usbhid [ 3.486856] usbhid: USB HID core driver [ 3.488370] drop_monitor: Initializing network drop monitor service [ 3.490443] Initializing XFRM netlink socket [ 3.492243] NET: Registered protocol family 10 [ 3.495399] Segment Routing with IPv6 [ 3.496971] NET: Registered protocol family 17 [ 3.499574] mpls_gso: MPLS GSO support [ 3.505155] RAS: Correctable Errors collector initialized. [ 3.507720] AVX version of gcm_enc/dec engaged. [ 3.509575] AES CTR mode by8 optimization enabled [ 3.588113] sched_clock: Marking stable (3588100373, 0)->(4525967482, -937867109) [ 3.592864] registered taskstats version 1 [ 3.595222] Loading compiled-in X.509 certificates [ 3.597506] zswap: loaded using pool lzo/zbud [ 3.624297] Key type big_key registered [ 3.636550] Key type encrypted registered [ 3.646350] ima: No TPM chip found, activating TPM-bypass! [ 3.648505] ima: Allocated hash algorithm: sha1 [ 3.650079] ima: No architecture policies found [ 3.653086] evm: Initialising EVM extended attributes: [ 3.654808] evm: security.selinux [ 3.655885] evm: security.ima [ 3.656755] evm: security.capability [ 3.657760] evm: HMAC attrs: 0x1 [ 3.662093] rtc_cmos 00:05: setting system clock to 2026-09-30 12:55:43 UTC (1790772943) [ 3.669748] debug: unmapping init [mem 0xffffffffa1a03000-0xffffffffa1bfffff] [ 3.673817] debug: unmapping init [mem 0xffffffffa0782000-0xffffffffa0a58fff] [ 3.682199] Write protecting the kernel read-only data: 28672k [ 3.686315] debug: unmapping init [mem 0xffffffff9ee03000-0xffffffff9effffff] [ 3.689620] debug: unmapping init [mem 0xffffffff9f714000-0xffffffff9f7fffff] [ 3.728753] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.739059] systemd[1]: Detected virtualization kvm. [ 3.741239] systemd[1]: Detected architecture x86-64. [ 3.743493] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.771576] systemd[1]: No hostname configured. [ 3.774029] systemd[1]: Set hostname to . [ 3.776390] random: systemd: uninitialized urandom read (16 bytes read) [ 3.779426] systemd[1]: Initializing machine ID from random generator. [ 3.904900] random: systemd: uninitialized urandom read (16 bytes read) [ 3.908255] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.913303] random: systemd: uninitialized urandom read (16 bytes read) [ 3.916564] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.922218] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Journal Service... Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.517431] device-mapper: uevent: version 1.0.3 [ 4.519507] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ 5.037139] random: fast init done [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.226746] virtio_net virtio0 ens2: renamed from eth0 [ 5.256735] scsi host0: ata_piix [ 5.261908] scsi host1: ata_piix [ 5.263972] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.268671] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.168809] random: crng init done [ 10.175075] random: 7 urandom warning(s) missed due to ratelimiting [ 10.218687] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.687661] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.408841] printk: systemd: 24 output lines suppressed due to ratelimiting [ 15.355572] SELinux: Disabled at runtime. [ 15.498826] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.528310] systemd[1]: Detected virtualization kvm. [ 15.533887] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 17.427728] systemd[1]: initrd-switch-root.service: Succeeded. [ 17.434786] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 17.466158] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 17.474914] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 17.484371] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 17.503410] systemd[1]: Starting Journal Service... Starting Journal Service... [ 17.524982] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ 17.959469] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting udev Coldplug all Devices... [ OK ] Reached target Paths. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 18.705690] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 19.399965] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 19.518025] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 19.916379] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.971277] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (6s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit)[ 24.716758] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (7s / no limit)[ 25.377197] NFS: Registering the id_resolver key type [ 25.380604] Key type id_resolver registered [ 25.383073] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg329-server login: [ 43.771006] hrtimer: interrupt took 7960814 ns [ 85.573473] libcfs: loading out-of-tree module taints kernel. [ 85.605840] Key type ._llcrypt registered [ 85.607852] Key type .llcrypt registered [ 85.685737] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 103.410452] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 105.282028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 105.324610] alg: No test for adler32 (adler32-zlib) [ 107.087467] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 108.383569] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 110.210229] Key type lgssc registered [ 112.517570] Lustre: Echo OBD driver; http://www.lustre.org/ [ 132.528055] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 174.251856] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 187.852471] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 187.915727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 189.246929] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 189.286431] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 189.377843] Lustre: lustre-MDT0000: new disk, initializing [ 189.489507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 189.528752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 194.390764] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 208.633971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 208.769637] Lustre: 6517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 208.842947] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 208.851368] Lustre: Skipped 1 previous similar message [ 208.978535] Lustre: lustre-MDT0001: new disk, initializing [ 209.065892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 209.120342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 209.143894] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 213.647933] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 218.862574] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 228.492183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 228.867779] Lustre: lustre-OST0000: new disk, initializing [ 228.875568] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 228.881771] Lustre: 8456:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 228.961786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 235.859407] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.139115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 239.150986] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 239.238363] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 250.820392] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 251.004685] Lustre: lustre-OST0001: new disk, initializing [ 251.008281] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 251.014234] Lustre: 9529:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 251.091980] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 257.829607] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 258.074537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 258.085696] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 258.178161] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 268.372482] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 279.832584] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 287.230122] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing check_logdir /tmp/testlogs/ [ 291.847754] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing yml_node [ 295.675746] Lustre: DEBUG MARKER: Client: 2.17.59.4 [ 298.230197] Lustre: DEBUG MARKER: MDS: 2.17.59.4 [ 301.353602] Lustre: DEBUG MARKER: OSS: 2.17.59.4 [ 303.051890] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Sep 30 09:00:40 EDT 2026 [ 321.674453] Lustre: DEBUG MARKER: only running tests: 76-120 [ 323.505209] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 325.343448] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 342.496586] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 342.502250] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 342.524560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 345.058172] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 345.069516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 345.081703] Lustre: Skipped 2 previous similar messages [ 345.089410] Lustre: Skipped 2 previous similar messages [ 346.781620] Lustre: server umount lustre-MDT0000 complete [ 350.215457] LustreError: 8455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790773290 with bad export cookie 9507269635588274983 [ 350.220034] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 350.227103] LustreError: 8455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 355.296204] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 355.296804] LustreError: 6524:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 355.299817] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 355.299826] Lustre: Skipped 1 previous similar message [ 355.312525] Lustre: Skipped 1 previous similar message [ 355.329591] LustreError: 6524:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 356.573841] Lustre: server umount lustre-MDT0001 complete [ 360.318760] Lustre: server umount lustre-OST0000 complete [ 363.539321] Lustre: server umount lustre-OST0001 complete [ 369.491427] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 377.818392] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 409.943518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 410.327515] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 410.393412] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 410.499348] Lustre: lustre-MDT0000: new disk, initializing [ 410.645847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 410.682191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 415.866863] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 425.641390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 425.784462] Lustre: 15139:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 425.855293] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 425.859540] Lustre: Skipped 1 previous similar message [ 425.972237] Lustre: lustre-MDT0001: new disk, initializing [ 426.112218] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 426.148644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 426.161925] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 430.194466] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 438.666747] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 443.405852] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 448.793431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 449.098507] Lustre: lustre-OST0000: new disk, initializing [ 449.103940] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 449.113560] Lustre: 17109:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 449.178723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 450.245371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 450.258478] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 450.394065] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 454.348115] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 463.133649] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 468.830319] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 469.080084] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 473.770509] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 473.977510] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 481.249362] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 481.251221] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 481.261372] LustreError: Skipped 1 previous similar message [ 481.266527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 481.284190] Lustre: Skipped 1 previous similar message [ 481.290240] Lustre: Skipped 1 previous similar message [ 482.482135] Lustre: server umount lustre-OST0000 complete [ 486.371570] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 486.398726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 486.404044] Lustre: Skipped 2 previous similar messages [ 492.226655] Lustre: server umount lustre-MDT0000 complete [ 495.807050] LustreError: 15132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790773435 with bad export cookie 9507269635588276509 [ 495.817360] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 495.835730] LustreError: 15132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 496.129832] Lustre: server umount lustre-MDT0001 complete [ 500.557635] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 09:03:58 (1790773438) [ 507.549457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 508.310266] LustreError: 19374:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 508.409389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 512.708288] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 513.507527] LustreError: 19375:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 518.637585] LustreError: 19374:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 521.389928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 526.217495] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 532.582236] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 537.927841] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 544.579143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 550.861681] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 557.831082] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 561.168996] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 567.777376] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 567.783137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 567.801020] Lustre: Skipped 1 previous similar message [ 567.807438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 567.812144] Lustre: Skipped 2 previous similar messages [ 573.925300] LustreError: 19374:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 573.947366] LustreError: 19374:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 573.963298] Lustre: server umount lustre-MDT0000 complete [ 577.594235] LustreError: 19356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790773517 with bad export cookie 9507269635588277398 [ 577.609542] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 577.985754] Lustre: server umount lustre-MDT0001 complete [ 591.459295] Lustre: server umount lustre-OST0000 complete [ 614.460689] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 625.109591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 625.544222] LustreError: 23757:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 625.637256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 625.645325] Lustre: Skipped 2 previous similar messages [ 630.075308] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 638.013975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 642.944436] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 645.852271] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 653.014856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 660.220710] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 670.460980] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 670.647520] Lustre: lustre-OST0001: new disk, initializing [ 670.657124] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 670.680382] Lustre: 26076:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 671.981167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 671.998240] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 672.131608] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 678.064473] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 686.880358] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 690.208274] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 702.945556] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 702.947811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 702.962673] Lustre: Skipped 4 previous similar messages [ 702.981263] Lustre: Skipped 6 previous similar messages [ 706.999856] Lustre: server umount lustre-MDT0000 complete [ 708.086180] LustreError: 24488:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 708.096819] LustreError: 24488:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 710.646279] LustreError: 26075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790773650 with bad export cookie 9507269635588278238 [ 710.670779] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 710.673829] LustreError: 26075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 710.975760] Lustre: server umount lustre-MDT0001 complete [ 725.102449] Lustre: server umount lustre-OST0000 complete [ 739.226170] Lustre: server umount lustre-OST0001 complete [ 754.565406] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 763.930293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 764.394996] LustreError: 29096:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 764.522452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 764.532366] Lustre: Skipped 3 previous similar messages [ 768.261744] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 776.500455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 781.187566] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 784.255742] Lustre: 30243:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 791.124208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 797.489376] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 805.921055] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 812.331185] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 820.318511] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 824.589911] Lustre: 32125:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 832.992772] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 833.010651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 833.020135] Lustre: Skipped 1 previous similar message [ 833.023970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 837.698426] Lustre: server umount lustre-MDT0000 complete [ 840.177424] LustreError: 29831:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 840.192978] LustreError: 29831:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 841.465252] LustreError: 29076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790773781 with bad export cookie 9507269635588279533 [ 841.468567] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 841.483736] LustreError: 29076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 841.839365] Lustre: server umount lustre-MDT0001 complete [ 854.633813] Lustre: server umount lustre-OST0000 complete [ 867.546848] Lustre: server umount lustre-OST0001 complete [ 875.171806] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 09:10:13 (1790773813) [ 899.909170] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 908.547369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 913.209452] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 921.852540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 926.287853] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 929.570488] Lustre: 36075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 936.052387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 941.584823] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 949.999953] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 957.629250] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 966.899406] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 970.559579] Lustre: 37954:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 978.405535] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 978.413130] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 978.426860] Lustre: Skipped 3 previous similar messages [ 978.435451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 978.439799] Lustre: Skipped 4 previous similar messages [ 982.534709] Lustre: server umount lustre-MDT0000 complete [ 984.038713] LustreError: 34921:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 984.057231] LustreError: 34921:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 985.924579] LustreError: 34907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790773925 with bad export cookie 9507269635588280646 [ 985.928953] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 985.943308] LustreError: 34907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 986.204990] Lustre: server umount lustre-MDT0001 complete [ 999.848211] Lustre: server umount lustre-OST0000 complete [ 1013.743685] Lustre: server umount lustre-OST0001 complete [ 1025.438270] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 09:12:43 (1790773963) [ 1041.372486] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 1054.037919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1054.407585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1054.414798] Lustre: Skipped 7 previous similar messages [ 1059.505835] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1070.767097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1074.548799] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1077.262240] Lustre: 41260:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1083.849796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1089.759445] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1104.911980] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1111.318926] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1118.906773] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1122.245924] Lustre: 43138:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1123.253272] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1123.262941] Lustre: Skipped 1 previous similar message [ 1127.392757] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1127.396223] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1127.401822] Lustre: Skipped 3 previous similar messages [ 1127.418701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1127.420903] Lustre: Skipped 3 previous similar messages [ 1132.257213] Lustre: server umount lustre-MDT0000 complete [ 1135.452300] LustreError: 40091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790774075 with bad export cookie 9507269635588281745 [ 1135.464595] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1141.886900] Lustre: server umount lustre-MDT0001 complete [ 1151.033874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1160.672462] LustreError: 3647:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8d0d03b41c00 x1877761639974400/t0(0) o250->MGC192.168.203.129@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1165.990647] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1170.361854] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1175.470629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1180.802421] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1181.170939] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1181.174939] Lustre: Skipped 1 previous similar message [ 1188.737724] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1190.485784] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1197.069618] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1198.453210] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1206.774778] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1209.791636] Lustre: server umount lustre-MDT0000 complete [ 1213.827654] LustreError: 40091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790774153 with bad export cookie 9507269635588282851 [ 1213.832350] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1213.835390] LustreError: 40091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1214.227114] Lustre: server umount lustre-MDT0001 complete [ 1229.202160] Lustre: server umount lustre-OST0000 complete [ 1232.351552] Lustre: 3651:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790774156/real 1790774156] req@ffff8d0e2e3b1180 x1877761640009600/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790774172 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1232.943110] Lustre: server umount lustre-OST0001 complete [ 1241.255246] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 09:16:18 (1790774178) [ 1257.335351] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 1268.042653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1268.452936] LustreError: 47743:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1268.473395] LustreError: 47743:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 30 previous similar messages [ 1273.238254] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1281.489524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1287.404727] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1290.278981] Lustre: 48895:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1296.528585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1305.123119] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1314.233211] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1321.195229] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1328.414152] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1331.777810] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1350.243122] Lustre: server umount lustre-MDT0000 complete [ 1355.197593] LustreError: 47725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790774295 with bad export cookie 9507269635588283670 [ 1355.199113] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1355.237739] LustreError: 47725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1355.682996] Lustre: server umount lustre-MDT0001 complete [ 1370.384512] Lustre: server umount lustre-OST0000 complete [ 1372.131827] Lustre: 3648:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790774295/real 1790774295] req@ffff8d0e352cb480 x1877761640066304/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790774311 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1375.154841] Lustre: server umount lustre-OST0001 complete [ 1386.289877] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 09:18:43 (1790774323) [ 1389.759840] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1392.052571] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 09:18:49 (1790774329) [ 1404.339301] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 1414.408766] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 1439.725648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1439.955973] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1440.044473] Lustre: lustre-MDT0000: new disk, initializing [ 1440.116643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1444.477543] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1454.283841] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1463.716712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1464.003249] Lustre: lustre-OST0000: new disk, initializing [ 1464.006612] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1464.009832] Lustre: Skipped 1 previous similar message [ 1464.013835] Lustre: 54972:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1465.548793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1465.559409] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1465.588956] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1470.196728] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1480.013387] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1482.211221] Lustre: DEBUG MARKER: create test files [ 1517.028484] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1517.050531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1517.080718] Lustre: Skipped 15 previous similar messages [ 1517.100403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1517.104514] Lustre: Skipped 18 previous similar messages [ 1517.779779] Lustre: server umount lustre-OST0000 complete [ 1522.158227] LustreError: 3647:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8d0d0878fb80 x1877761640085632/t0(0) o8->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1522.827371] Lustre: server umount lustre-MDT0000 complete [ 1536.565380] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 1540.792673] Key type lgssc unregistered [ 1541.320672] LNet: 56601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1541.329600] LNetError: 56601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1541.364560] LNet: Removed LNI 192.168.203.129@tcp [ 1543.325154] Key type .llcrypt unregistered [ 1543.326958] Key type ._llcrypt unregistered [ 1574.264580] Key type ._llcrypt registered [ 1574.266627] Key type .llcrypt registered [ 1574.375763] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 1575.058631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1575.247516] alg: No test for adler32 (adler32-zlib) [ 1576.355738] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 1576.683258] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 1578.387609] Key type lgssc registered [ 1579.654671] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1587.731849] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1587.780312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1589.336116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1593.584928] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1600.998250] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1607.582156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1608.073026] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1614.211344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1614.301653] Lustre: 59351:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 511 < left 544, rollback = 9 [ 1614.323949] Lustre: 59351:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 32/128/4, destroy: 0/0/0 [ 1614.336824] Lustre: 59351:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1614.341192] Lustre: 59351:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1614.345524] Lustre: 59351:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1614.354041] Lustre: 59351:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1614.971337] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1622.883241] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1628.456307] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1634.232274] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1645.026956] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1645.038140] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1645.052500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1647.606475] Lustre: server umount lustre-OST0000 complete [ 1651.898766] Lustre: server umount lustre-MDT0000 complete [ 1664.619569] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 1668.202432] Key type lgssc unregistered [ 1668.538091] LNet: 60962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1668.554452] LNetError: 60962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1668.577216] LNet: Removed LNI 192.168.203.129@tcp [ 1669.631394] Key type .llcrypt unregistered [ 1669.633400] Key type ._llcrypt unregistered [ 1704.968571] Key type ._llcrypt registered [ 1704.970660] Key type .llcrypt registered [ 1705.055164] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 1706.096558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1706.219583] alg: No test for adler32 (adler32-zlib) [ 1707.578786] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 1707.970339] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 1709.703355] Key type lgssc registered [ 1711.437408] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1720.029203] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1720.079666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1721.556276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1726.082725] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1734.534727] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1741.564046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1742.021825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1747.145804] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1747.252462] Lustre: 63828:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 510 < left 544, rollback = 9 [ 1747.260350] Lustre: 63828:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 32/128/5, destroy: 0/0/0 [ 1747.267728] Lustre: 63828:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1747.274952] Lustre: 63828:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1747.285366] Lustre: 63828:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1747.292879] Lustre: 63828:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1748.972932] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1757.408776] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1762.631573] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1773.029253] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1773.036417] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1773.069216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1778.148981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1778.341292] Lustre: server umount lustre-OST0000 complete [ 1782.707904] Lustre: server umount lustre-MDT0000 complete [ 1795.255479] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 1798.559977] Key type lgssc unregistered [ 1798.939400] LNet: 65384:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1798.948757] LNetError: 65384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1799.974703] LNet: Removed LNI 192.168.203.129@tcp [ 1801.185452] Key type .llcrypt unregistered [ 1801.189541] Key type ._llcrypt unregistered [ 1821.567631] Key type ._llcrypt registered [ 1821.571155] Key type .llcrypt registered [ 1821.739340] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 1838.678118] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 1840.014723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1840.170766] alg: No test for adler32 (adler32-zlib) [ 1841.388853] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 1841.676840] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 1843.391216] Key type lgssc registered [ 1844.618248] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1882.101299] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1882.134291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1883.508875] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1883.553193] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1883.666922] Lustre: lustre-MDT0000: new disk, initializing [ 1883.759159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1883.774667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1888.717150] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1900.492778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1900.649627] Lustre: 68553:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1900.779652] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1900.791966] Lustre: Skipped 1 previous similar message [ 1900.953250] Lustre: lustre-MDT0001: new disk, initializing [ 1901.110377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1901.167298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1901.195442] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1906.116672] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1916.940451] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1924.908323] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1935.146551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1935.686497] Lustre: lustre-OST0000: new disk, initializing [ 1935.692099] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1935.705966] Lustre: 70525:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1935.811419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1941.094196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1941.101505] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1941.255079] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1943.297736] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1953.834807] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1959.231503] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1959.525678] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1964.678229] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1964.867993] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1972.192867] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1972.197913] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1972.207378] LustreError: Skipped 1 previous similar message [ 1972.209815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1972.234325] Lustre: Skipped 1 previous similar message [ 1972.249204] Lustre: Skipped 1 previous similar message [ 1973.493525] Lustre: server umount lustre-OST0000 complete [ 1977.313426] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1977.326066] Lustre: Skipped 1 previous similar message [ 1977.328256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1977.335127] Lustre: Skipped 1 previous similar message [ 1982.433807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1982.442739] Lustre: Skipped 1 previous similar message [ 1983.515163] Lustre: server umount lustre-MDT0000 complete [ 1987.442484] LustreError: 68544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790774927 with bad export cookie 7146335294116477275 [ 1987.446384] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1987.457741] LustreError: 68544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1987.744586] Lustre: server umount lustre-MDT0001 complete [ 1996.138321] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 09:28:53 (1790774933) [ 2007.363758] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 2045.679868] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 2055.636090] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 2097.673207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2097.938721] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2097.967876] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2098.105881] Lustre: lustre-MDT0000: new disk, initializing [ 2098.174732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2098.194646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2103.084684] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2115.030658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2115.193202] Lustre: 76544:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2115.280335] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2115.288303] Lustre: Skipped 1 previous similar message [ 2115.380659] Lustre: lustre-MDT0001: new disk, initializing [ 2115.497447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2115.565061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2115.572706] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2119.866643] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2129.524572] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2135.484211] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2143.010624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2143.396451] Lustre: lustre-OST0000: new disk, initializing [ 2143.403282] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2143.415204] Lustre: 78513:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2143.518402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2149.983220] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2150.994104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2151.016793] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2151.176169] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2161.093756] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2167.193609] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2167.594288] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2173.607515] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2173.896850] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2176.994891] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2176.995665] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2177.001220] LustreError: Skipped 1 previous similar message [ 2177.022848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2177.031916] Lustre: Skipped 1 previous similar message [ 2177.062266] Lustre: Skipped 1 previous similar message [ 2185.188430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2185.195537] Lustre: Skipped 3 previous similar messages [ 2190.303711] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2190.519504] Lustre: server umount lustre-OST0000 complete [ 2195.433778] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2195.436326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2195.461311] Lustre: Skipped 1 previous similar message [ 2195.475750] Lustre: Skipped 3 previous similar messages [ 2200.410665] Lustre: server umount lustre-MDT0000 complete [ 2204.350542] LustreError: 76537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790775144 with bad export cookie 7146335294116478150 [ 2204.352318] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2204.364497] LustreError: 76537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2204.808816] Lustre: server umount lustre-MDT0001 complete [ 2213.479221] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 09:32:31 (1790775151) [ 2219.715057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2220.189297] LustreError: 80926:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2220.220384] LustreError: 80926:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2220.299572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2224.660716] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2225.638447] LustreError: 80927:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2230.758807] LustreError: 80926:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2233.450517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2238.317222] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2245.145172] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2250.587666] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2257.903588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2258.415652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2258.435897] Lustre: Skipped 1 previous similar message [ 2265.481303] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2274.618104] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2279.051031] Lustre: 83601:0:(genops.c:1773:obd_export_evict_by_uuid()) MGS: evicting 6e4b0934-2ac8-49d5-ac7e-b26225594172 at adminstrative request [ 2280.943408] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2280.961562] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2291.182291] Lustre: MGS: Client 6e4b0934-2ac8-49d5-ac7e-b26225594172 (at 0@lo) reconnecting [ 2291.190396] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x632ce2164ba6403f to 0x632ce2164ba642df [ 2291.206626] Lustre: MGC192.168.203.129@tcp: Connection restored to 0@lo (at 0@lo) [ 2316.358931] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2316.546945] Lustre: lustre-OST0001: new disk, initializing [ 2316.551748] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2316.558182] Lustre: 83997:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2322.054724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2322.063204] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2322.205347] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2322.706519] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2332.549480] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2337.764322] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2337.771704] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2337.778558] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2341.602536] Lustre: server umount lustre-OST0001 complete [ 2342.889385] LustreError: 82748:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2342.916107] LustreError: 82748:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2346.466224] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2346.478518] LustreError: Skipped 1 previous similar message [ 2346.483226] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2346.496722] Lustre: Skipped 1 previous similar message [ 2348.002634] LustreError: 83636:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2348.040930] LustreError: 83636:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2351.734446] Lustre: server umount lustre-OST0000 complete [ 2356.706324] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2356.718869] LustreError: Skipped 1 previous similar message [ 2362.179232] Lustre: server umount lustre-MDT0000 complete [ 2363.366148] LustreError: 81941:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2363.382472] LustreError: 81941:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2366.313024] LustreError: 80907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790775306 with bad export cookie 7146335294116479711 [ 2366.317047] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2366.329437] LustreError: 80907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2366.837428] Lustre: server umount lustre-MDT0001 complete [ 2375.664302] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 09:35:13 (1790775313) [ 2377.144451] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2378.903498] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 09:35:16 (1790775316) [ 2380.512693] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2382.722828] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 09:35:19 (1790775319) [ 2384.565533] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2386.511385] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 09:35:24 (1790775324) [ 2388.557827] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2390.672461] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 09:35:28 (1790775328) [ 2392.835725] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2399.166872] loop: module loaded [ 2399.199459] loop0: detected capacity change from 0 to 4294967296 [ 2399.274115] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2400.813460] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2404.695136] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2409.022577] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2435.201884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2435.250645] LustreError: 88117:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2435.253829] LustreError: 88117:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2435.257642] LustreError: 88117:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2435.264338] LustreError: 88117:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2456.293667] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 2467.886268] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 2513.882520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2514.163023] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2514.230301] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2514.382759] Lustre: lustre-MDT0000: new disk, initializing [ 2514.477430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2514.492688] Lustre: Skipped 1 previous similar message [ 2514.532530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2519.270815] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2532.535394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2532.631054] Lustre: 90986:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2533.103421] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2539.067946] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2549.183754] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2554.758087] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2561.843163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2562.106501] Lustre: 92954:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2568.856363] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2569.834490] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2569.989261] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2579.855858] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2585.635074] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2585.994222] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2591.493598] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2591.820838] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2595.809977] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2595.813643] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2595.832091] LustreError: Skipped 1 previous similar message [ 2595.835568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2595.871961] Lustre: Skipped 4 previous similar messages [ 2595.915600] Lustre: Skipped 8 previous similar messages [ 2608.095151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2608.342670] Lustre: server umount lustre-OST0000 complete [ 2615.264983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2618.696558] Lustre: server umount lustre-MDT0000 complete [ 2621.415323] LustreError: 90991:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2621.452853] LustreError: 90991:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2622.852265] LustreError: 90978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790775562 with bad export cookie 7146335294116480327 [ 2622.853931] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2622.863193] LustreError: 90978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2623.230946] Lustre: server umount lustre-MDT0001 complete [ 2633.619933] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 09:39:30 (1790775570) [ 2640.376638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2645.893910] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2654.296081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2654.580097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-60 [ 2654.587816] Lustre: Skipped 3 previous similar messages [ 2660.279481] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2668.290837] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2674.535190] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2682.220836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2689.236189] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2697.071026] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2701.662932] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2701.862097] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2706.322931] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2706.558730] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2713.232315] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2713.385550] Lustre: lustre-OST0001: new disk, initializing [ 2713.389175] Lustre: Skipped 2 previous similar messages [ 2713.394586] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2713.399348] Lustre: Skipped 3 previous similar messages [ 2713.404053] Lustre: 98740:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2719.961671] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2720.877412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2720.892851] Lustre: Skipped 2 previous similar messages [ 2720.899130] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2721.009966] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2730.044933] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2734.636447] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2734.840692] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2739.596126] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2739.798129] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2758.688979] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2784.168943] Lustre: Failing over lustre-MDT0000 [ 2784.581653] Lustre: server umount lustre-MDT0000 complete [ 2787.817344] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2787.826634] Lustre: Skipped 2 previous similar messages [ 2787.831891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2787.832563] LustreError: 96528:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2787.832569] LustreError: 96528:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2804.196402] Lustre: 66767:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790775727/real 1790775727] req@00000000103e594a x1877763457041408/t0(0) o400->MGC192.168.203.129@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790775743 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2804.241234] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2809.104081] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2809.106186] LDISKFS-fs (dm-0): recovery complete [ 2809.116613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2814.445689] LustreError: 66764:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c36181ee x1877763457050624/t0(0) o250->MGC192.168.203.129@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2814.822845] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2818.960543] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2819.087222] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2819.087425] Lustre: 96528:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2820.080878] Lustre: 96099:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2820.098081] Lustre: 96099:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 1 previous similar message [ 2820.119167] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2820.218887] LustreError: 101588:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2820.536150] LustreError: 101588:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2820.873695] LustreError: 101588:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2820.891952] LustreError: 101588:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 1 previous similar message [ 2821.215122] LustreError: 101588:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2821.224801] LustreError: 101588:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 1 previous similar message [ 2821.232495] Lustre: 101588:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2821.252799] Lustre: 101588:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 7 previous similar messages [ 2821.904805] LustreError: 101588:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2821.924331] LustreError: 101588:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 2 previous similar messages [ 2822.248131] LustreError: 101588:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2822.266727] LustreError: 101588:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 2 previous similar messages [ 2823.233944] Lustre: 101588:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2823.243710] Lustre: 101588:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 5 previous similar messages [ 2824.212463] LustreError: 101588:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2824.229042] LustreError: 101588:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 6 previous similar messages [ 2824.557322] LustreError: 101588:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2824.572045] LustreError: 101588:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 6 previous similar messages [ 2827.477143] Lustre: 101588:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2827.502077] Lustre: 101588:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 12 previous similar messages [ 2828.464192] LustreError: 101588:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2828.476128] LustreError: 101588:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 12 previous similar messages [ 2828.799165] LustreError: 101588:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2828.813317] LustreError: 101588:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 12 previous similar messages [ 2835.513719] Lustre: 101588:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2835.538707] Lustre: 101588:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 24 previous similar messages [ 2836.494905] LustreError: 101588:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2836.514849] LustreError: 101588:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 24 previous similar messages [ 2836.839170] LustreError: 101588:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2836.851097] LustreError: 101588:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 24 previous similar messages [ 2851.826277] Lustre: 101588:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2851.853958] Lustre: 101588:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 50 previous similar messages [ 2852.800769] LustreError: 101588:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2852.811109] LustreError: 101588:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 50 previous similar messages [ 2853.127129] LustreError: 101588:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2853.133454] LustreError: 101588:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 50 previous similar messages [ 2879.168075] Lustre: 101588:0:(ldlm_lib.c:2428:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2879.177933] Lustre: 101588:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2879.186954] Lustre: 101588:0:(genops.c:1600:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client bdcb18f4-034c-415d-bf3e-3437a55b8c0f@192.168.203.29@tcp [ 2879.214932] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2879.223610] Lustre: 101588:0:(ldlm_lib.c:1948:abort_req_replay_queue()) @@@ aborted: req@00000000be540380 x1877763437958144/t0(8589934781) o101->bdcb18f4-034c-415d-bf3e-3437a55b8c0f@192.168.203.29@tcp:389/0 lens 592/0 e 0 to 0 dl 1790775829 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2879.256624] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2879.295397] Lustre: lustre-MDT0000: Denying connection for new client bdcb18f4-034c-415d-bf3e-3437a55b8c0f (at 192.168.203.29@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 2879.306411] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2879.352972] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2879.374269] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2879.376448] Lustre: Skipped 2 previous similar messages [ 2879.438660] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:80 to 0x280000401:577) [ 2879.439847] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:48 to 0x2c0000401:545) [ 2896.872976] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2896.883938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2896.892611] Lustre: Skipped 3 previous similar messages [ 2896.916887] Lustre: Skipped 8 previous similar messages [ 2911.199394] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2911.399669] Lustre: server umount lustre-OST0000 complete [ 2912.228315] LustreError: 97193:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2912.255494] LustreError: 97193:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 42 previous similar messages [ 2915.810145] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2921.466170] Lustre: server umount lustre-OST0001 complete [ 2931.594662] Lustre: server umount lustre-MDT0000 complete [ 2935.142490] LustreError: 98739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790775874 with bad export cookie 7146335294116531546 [ 2935.148040] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2935.154592] LustreError: 98739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2935.454953] Lustre: server umount lustre-MDT0001 complete [ 2944.056866] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 09:44:41 (1790775881) [ 2952.163287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2952.171159] Lustre: *** cfs_fail_loc=197, val=0*** [ 2967.777298] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2972.897169] LustreError: 103566:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.129@tcp: failed processing log, type 4: rc = -110 [ 2978.043717] Lustre: *** cfs_fail_loc=197, val=0*** [ 2978.048303] Lustre: Skipped 65 previous similar messages [ 2998.624348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2998.633876] Lustre: Skipped 3 previous similar messages [ 3004.671690] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3011.807852] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3013.913781] Lustre: server umount lustre-OST0000 complete [ 3023.225989] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 09:46:00 (1790775960) [ 3047.139663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3048.205557] LustreError: 105186:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3048.236526] LustreError: 105186:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3055.359253] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3068.012677] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3072.454801] LustreError: 105217:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3072.471165] LustreError: 105217:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3072.479943] LustreError: 105217:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 24, retries 0, failed: rc = -5 [ 3072.815673] Lustre: server umount lustre-MDT0000 complete [ 3109.978759] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 3121.475825] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 3162.270698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3162.459577] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3162.491321] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3162.583384] Lustre: lustre-MDT0000: new disk, initializing [ 3162.687656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3168.243737] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3180.415464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3180.582439] Lustre: 109393:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3180.955681] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3186.001130] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3197.220732] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3204.565476] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3212.668302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3212.897510] Lustre: 111364:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3220.501166] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3221.912848] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3232.526599] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3238.550466] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3238.921478] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3244.596729] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3244.887766] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3247.583944] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3247.594171] LustreError: Skipped 2 previous similar messages [ 3247.596555] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3247.608943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3247.619884] Lustre: Skipped 5 previous similar messages [ 3247.640670] Lustre: Skipped 11 previous similar messages [ 3253.395888] Lustre: server umount lustre-OST0000 complete [ 3272.159880] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3272.765382] Lustre: server umount lustre-MDT0000 complete [ 3277.083988] LustreError: 109385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790776216 with bad export cookie 7146335294116542242 [ 3277.085435] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3277.096515] LustreError: 109385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3277.621562] Lustre: server umount lustre-MDT0001 complete [ 3288.667403] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 09:50:25 (1790776225) [ 3300.369496] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 3303.901620] Key type lgssc unregistered [ 3304.275241] LNet: 113856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3304.300323] LNetError: 113856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3305.319976] LNet: Removed LNI 192.168.203.129@tcp [ 3306.261523] Key type .llcrypt unregistered [ 3306.263405] Key type ._llcrypt unregistered [ 3331.380069] Key type ._llcrypt registered [ 3331.383428] Key type .llcrypt registered [ 3331.547840] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 3351.359808] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 3352.619402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3352.721600] alg: No test for adler32 (adler32-zlib) [ 3353.889134] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 3354.220935] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 3355.903412] Key type lgssc registered [ 3357.359816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3401.045786] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3401.077378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3402.421970] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3402.483046] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3402.598993] Lustre: lustre-MDT0000: new disk, initializing [ 3402.706721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3402.740781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3408.492648] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3422.516957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3422.818292] Lustre: lustre-OST0000: new disk, initializing [ 3422.821451] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3422.827068] Lustre: Skipped 1 previous similar message [ 3422.831907] Lustre: 118048:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3422.891317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3430.417291] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3432.062428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3432.070670] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3432.088428] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3441.548781] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3441.756547] Lustre: lustre-OST0001: new disk, initializing [ 3441.762508] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3441.771223] Lustre: 118912:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3441.840679] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3447.548619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3447.569483] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3447.605396] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3449.438941] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3463.148073] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3463.157134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3463.161685] Lustre: Skipped 1 previous similar message [ 3463.175711] Lustre: Skipped 1 previous similar message [ 3464.975423] Lustre: server umount lustre-MDT0000 complete [ 3471.941389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3476.192049] LustreError: 117233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790776416 with bad export cookie 611893116555344556 [ 3476.202198] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3476.206047] LustreError: 117233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3476.336043] Lustre: server umount lustre-OST0000 complete [ 3479.923530] Lustre: server umount lustre-OST0001 complete [ 3489.206853] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 09:53:46 (1790776426) [ 3516.322905] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 09:54:13 (1790776453) [ 3528.286750] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 3551.436467] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 09:54:49 (1790776489) [ 3590.586194] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 3605.641742] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 3661.827127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3662.168549] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3662.222641] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3662.315103] Lustre: lustre-MDT0000: new disk, initializing [ 3662.481395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3662.522767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3669.641226] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3682.277223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3682.492741] Lustre: 126673:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3682.584248] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3682.589728] Lustre: Skipped 1 previous similar message [ 3682.740437] Lustre: lustre-MDT0001: new disk, initializing [ 3682.885825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3682.961067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3682.979567] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3686.971134] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3697.212512] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3702.208266] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3708.515432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3708.712701] Lustre: lustre-OST0000: new disk, initializing [ 3708.717617] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3708.728586] Lustre: 128642:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3708.795184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3714.007039] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3718.193785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3718.205235] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3718.335352] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3722.326448] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3726.433834] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3726.626836] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3730.443516] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3730.667976] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3733.991370] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3733.996065] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3733.998808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3734.032846] Lustre: Skipped 1 previous similar message [ 3738.541073] Lustre: server umount lustre-OST0000 complete [ 3744.226971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3744.228597] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3744.244248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3744.244260] Lustre: Skipped 1 previous similar message [ 3744.288696] Lustre: Skipped 1 previous similar message [ 3749.124706] Lustre: server umount lustre-MDT0000 complete [ 3752.870139] LustreError: 126663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790776692 with bad export cookie 611893116555345781 [ 3752.876200] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3752.881620] LustreError: 126663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3753.257213] Lustre: server umount lustre-MDT0001 complete [ 3763.051977] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 09:58:20 (1790776700) [ 3771.306682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3772.097190] LustreError: 131052:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3772.133097] LustreError: 131052:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3772.196632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3777.507357] LustreError: 131053:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3778.502158] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3781.608192] LustreError: 131052:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3786.731618] LustreError: 131053:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3791.248945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3791.851397] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3791.869087] Lustre: Skipped 1 previous similar message [ 3798.143173] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3808.107195] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3814.994360] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3823.228849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3823.617980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3823.629464] Lustre: Skipped 1 previous similar message [ 3831.954303] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3843.421404] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3849.196902] Lustre: *** cfs_fail_loc=159, val=0*** [ 3849.209374] Lustre: Skipped 1 previous similar message [ 3865.665907] Lustre: lustre-MDT0000: Client a1533353-4847-4e68-8543-7e974090b43b (at 192.168.203.29@tcp) reconnecting [ 3867.371724] Lustre: *** cfs_fail_loc=159, val=0*** [ 3867.379388] Lustre: Skipped 2 previous similar messages [ 3883.077372] Lustre: lustre-MDT0000: Client a1533353-4847-4e68-8543-7e974090b43b (at 192.168.203.29@tcp) reconnecting [ 3903.468848] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3903.476178] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3903.484298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3903.516051] Lustre: Skipped 1 previous similar message [ 3906.709651] Lustre: server umount lustre-OST0000 complete [ 3913.699315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3913.710911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3913.718603] Lustre: Skipped 2 previous similar messages [ 3916.768379] Lustre: server umount lustre-MDT0000 complete [ 3918.820466] LustreError: 133588:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3918.840484] LustreError: 133588:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3920.319512] LustreError: 133657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790776860 with bad export cookie 611893116555346663 [ 3920.323413] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3920.336493] LustreError: 133657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3920.765300] Lustre: server umount lustre-MDT0001 complete [ 3931.168539] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 3933.768720] Key type lgssc unregistered [ 3934.054063] LNet: 134845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3934.066396] LNetError: 134845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3934.083927] LNet: Removed LNI 192.168.203.129@tcp [ 3935.076295] Key type .llcrypt unregistered [ 3935.079333] Key type ._llcrypt unregistered [ 3957.578370] Key type ._llcrypt registered [ 3957.581351] Key type .llcrypt registered [ 3957.706625] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 3958.727072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3958.748383] alg: No test for adler32 (adler32-zlib) [ 3959.842291] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 3960.074169] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 3961.791226] Key type lgssc registered [ 3962.921401] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3971.096417] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3971.113655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3972.662725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3977.983019] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3982.836550] LustreError: 136274:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3982.871498] LustreError: 136274:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3987.938664] LustreError: 136275:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3988.808205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3989.337225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3995.212910] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4003.791705] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4010.214205] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4018.057143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4018.625855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4025.530794] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4033.272488] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4043.197264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4051.974875] Lustre: *** cfs_fail_loc=159, val=0*** [ 4068.379190] Lustre: lustre-MDT0000: Client 35e11709-3b0c-4280-a9f7-ad8ce7aa50a1 (at 192.168.203.29@tcp) reconnecting [ 4070.652868] Lustre: *** cfs_fail_loc=159, val=0*** [ 4086.359991] Lustre: lustre-MDT0001: Client 35e11709-3b0c-4280-a9f7-ad8ce7aa50a1 (at 192.168.203.29@tcp) reconnecting [ 4088.408140] Lustre: *** cfs_fail_loc=159, val=0*** [ 4088.410624] Lustre: Skipped 3 previous similar messages [ 4104.770637] Lustre: lustre-MDT0001: Client 35e11709-3b0c-4280-a9f7-ad8ce7aa50a1 (at 192.168.203.29@tcp) reconnecting [ 4108.430792] Lustre: 139330:0:(mdt_handler.c:65:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4110.851193] Lustre: *** cfs_fail_loc=159, val=0*** [ 4110.865256] Lustre: Skipped 4 previous similar messages [ 4127.345768] Lustre: lustre-MDT0000: Client 656b28b2-eeec-49d9-9e1a-00455707dedb (at 192.168.203.29@tcp) reconnecting [ 4129.503373] Lustre: *** cfs_fail_loc=159, val=0*** [ 4129.506331] Lustre: Skipped 13 previous similar messages [ 4145.216326] Lustre: lustre-MDT0000: Client 656b28b2-eeec-49d9-9e1a-00455707dedb (at 192.168.203.29@tcp) reconnecting [ 4147.403384] Lustre: 139620:0:(mdt_handler.c:65:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4153.845982] LustreError: lustre-OST0000-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107 [ 4153.851684] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4153.858126] LustreError: Skipped 1 previous similar message [ 4153.891669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4155.879954] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4155.880146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4155.883678] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4155.883686] LustreError: Skipped 2 previous similar messages [ 4155.925788] Lustre: Skipped 1 previous similar message [ 4160.083331] Lustre: server umount lustre-OST0000 complete [ 4166.113895] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4166.116654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4166.154305] Lustre: Skipped 1 previous similar message [ 4166.179531] Lustre: Skipped 1 previous similar message [ 4170.172828] Lustre: server umount lustre-MDT0000 complete [ 4171.238467] LustreError: 138982:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4171.269273] LustreError: 138982:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4173.807623] LustreError: 139449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790777113 with bad export cookie 9330479298317737101 [ 4173.815334] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4173.824204] LustreError: 139449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4174.290995] Lustre: server umount lustre-MDT0001 complete [ 4187.414610] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 4190.440652] Key type lgssc unregistered [ 4190.805608] LNet: 140677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4190.811989] LNetError: 140677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4190.832178] LNet: Removed LNI 192.168.203.129@tcp [ 4192.008833] Key type .llcrypt unregistered [ 4192.012539] Key type ._llcrypt unregistered [ 4218.138435] Key type ._llcrypt registered [ 4218.143996] Key type .llcrypt registered [ 4218.246654] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 4219.208665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4219.325103] alg: No test for adler32 (adler32-zlib) [ 4220.477821] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 4220.759058] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 4222.506164] Key type lgssc registered [ 4223.738819] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4232.312475] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4232.358146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4233.920729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4238.517752] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4243.426666] LustreError: 142109:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4243.437775] LustreError: 142109:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4247.309534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4247.713690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4251.965863] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4259.208558] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4265.070229] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4271.889206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4272.197364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4273.269085] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4277.646537] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4278.395666] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4284.974954] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4291.899543] Lustre: 144693:0:(mdt_handler.c:65:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4295.267088] Lustre: 144742:0:(mdt_handler.c:65:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4296.692223] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4297.502381] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4301.348660] Lustre: 144889:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4302.165978] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4302.979845] Lustre: 144987:0:(mdt_handler.c:65:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4309.472345] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4309.474905] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4309.482723] LustreError: Skipped 1 previous similar message [ 4309.483862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4309.520730] Lustre: Skipped 1 previous similar message [ 4312.927590] Lustre: server umount lustre-OST0000 complete [ 4319.200614] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4319.209708] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4319.224825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4319.236132] Lustre: Skipped 1 previous similar message [ 4322.651762] Lustre: server umount lustre-MDT0000 complete [ 4324.833124] LustreError: 142109:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4324.867287] LustreError: 142109:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4326.568818] LustreError: 142091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790777266 with bad export cookie 9759530179311789846 [ 4326.574027] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4326.588233] LustreError: 142091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4326.891493] Lustre: server umount lustre-MDT0001 complete [ 4337.586089] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 4340.373937] Key type lgssc unregistered [ 4340.702382] LNet: 146047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4340.711981] LNetError: 146047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4340.734305] LNet: Removed LNI 192.168.203.129@tcp [ 4341.501335] Key type .llcrypt unregistered [ 4341.503188] Key type ._llcrypt unregistered [ 4367.198647] Key type ._llcrypt registered [ 4367.201648] Key type .llcrypt registered [ 4367.324916] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 4368.575665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4368.637493] alg: No test for adler32 (adler32-zlib) [ 4369.651864] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 4369.925203] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 4371.607906] Key type lgssc registered [ 4373.429509] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4382.019669] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4382.050316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4383.563764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4388.448685] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4393.964110] LustreError: 147477:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4393.981702] LustreError: 147477:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4397.888125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4398.448799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4403.403903] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4411.901255] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4418.309723] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4425.031117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4425.505566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4428.627710] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4432.522786] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4441.120425] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4445.054117] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4450.266115] Lustre: *** cfs_fail_loc=159, val=0*** [ 4466.764344] Lustre: lustre-MDT0000: Client daf6116f-d227-4db7-8588-45575a8ac18d (at 192.168.203.29@tcp) reconnecting [ 4472.819164] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4472.822611] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4472.824735] LustreError: Skipped 3 previous similar messages [ 4472.862412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4474.796872] Lustre: server umount lustre-OST0000 complete [ 4480.490872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4480.509671] LustreError: Skipped 3 previous similar messages [ 4480.519774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4480.555938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4486.114176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4486.119187] Lustre: Skipped 2 previous similar messages [ 4491.234305] Lustre: 146780:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790777415/real 1790777415] req@00000000fa5832a8 x1877766107597952/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790777431 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4491.237301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4491.285896] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4491.285905] Lustre: Skipped 1 previous similar message [ 4491.311895] Lustre: Skipped 1 previous similar message [ 4492.259026] Lustre: 146779:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790777415/real 1790777415] req@00000000d8128dee x1877766107598208/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1790777431 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4494.303160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4494.524635] Lustre: server umount lustre-MDT0000 complete [ 4496.352274] LustreError: 147476:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4496.387684] LustreError: 147476:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4496.416262] Lustre: 146779:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790777420/real 1790777420] req@000000002c6609ad x1877766107599616/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790777436 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4498.951897] LustreError: 147457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790777438 with bad export cookie 8639634491855871257 [ 4498.973613] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4498.975202] LustreError: 147457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4499.320714] Lustre: server umount lustre-MDT0001 complete [ 4511.732687] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 4514.073327] Key type lgssc unregistered [ 4514.369628] LNet: 151140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4514.382497] LNetError: 151140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4515.433079] LNet: Removed LNI 192.168.203.129@tcp [ 4516.379233] Key type .llcrypt unregistered [ 4516.381629] Key type ._llcrypt unregistered [ 4540.974771] Key type ._llcrypt registered [ 4540.976983] Key type .llcrypt registered [ 4541.081591] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 4542.384356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4542.464330] alg: No test for adler32 (adler32-zlib) [ 4543.684639] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 4544.077352] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 4545.807217] Key type lgssc registered [ 4546.953706] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4557.004856] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4557.093868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4559.105951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4565.276256] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4568.547499] LustreError: 152571:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4568.575280] LustreError: 152571:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4573.666215] LustreError: 152572:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4575.827653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4576.379181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4582.524536] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4591.355213] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4598.641299] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4606.583317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4606.903531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4613.749863] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4621.266562] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4622.755423] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4628.509963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4645.991478] Lustre: lustre-OST0000: evicting 2d6d7d98-b174-493f-9620-32d0bd03603f (at 192.168.203.29@tcp) by administrative request [ 4646.003987] Lustre: Skipped 1 previous similar message [ 4674.528579] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4674.533438] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4674.535436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4674.560017] Lustre: Skipped 1 previous similar message [ 4679.657591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4679.674236] Lustre: Skipped 2 previous similar messages [ 4680.533939] Lustre: server umount lustre-OST0000 complete [ 4684.769448] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4684.771841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4684.783850] Lustre: Skipped 1 previous similar message [ 4684.794935] Lustre: Skipped 1 previous similar message [ 4689.888063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4689.897628] Lustre: Skipped 1 previous similar message [ 4690.683509] Lustre: server umount lustre-MDT0000 complete [ 4694.333793] LustreError: 152552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790777634 with bad export cookie 4417293462289285249 [ 4694.349068] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4694.355956] LustreError: 152552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4694.663909] Lustre: server umount lustre-MDT0001 complete [ 4706.412506] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 4709.476409] Key type lgssc unregistered [ 4709.810824] LNet: 156642:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4709.822439] LNetError: 156642:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4709.832630] LNet: Removed LNI 192.168.203.129@tcp [ 4710.870604] Key type .llcrypt unregistered [ 4710.872643] Key type ._llcrypt unregistered [ 4790.688563] Key type ._llcrypt registered [ 4790.691860] Key type .llcrypt registered [ 4790.818947] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 4791.760379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4791.857702] alg: No test for adler32 (adler32-zlib) [ 4793.030505] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 4793.290287] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 4795.048384] Key type lgssc registered [ 4796.425407] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4804.754361] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4804.788630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4806.356271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4811.249355] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4816.877539] LustreError: 159144:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4816.892182] LustreError: 159144:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4820.266521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4820.738444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4826.272764] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4833.396088] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4838.647843] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4846.263473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4846.631275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4851.777506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 4853.578600] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4862.496465] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4866.118073] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 4876.777360] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4876.789381] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4876.825848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4881.381032] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4881.381244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4881.386405] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4881.417747] Lustre: Skipped 1 previous similar message [ 4882.606786] Lustre: server umount lustre-OST0000 complete [ 4886.496867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4886.498072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4886.525630] Lustre: Skipped 1 previous similar message [ 4886.535852] Lustre: Skipped 1 previous similar message [ 4891.616591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4891.629499] Lustre: Skipped 1 previous similar message [ 4892.871611] Lustre: server umount lustre-MDT0000 complete [ 4896.693493] LustreError: 159125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790777836 with bad export cookie 14440117184663903053 [ 4896.700991] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4897.020290] Lustre: server umount lustre-MDT0001 complete [ 4911.150914] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 4913.808383] Key type lgssc unregistered [ 4914.087589] LNet: 162709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4914.101593] LNetError: 162709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4914.127850] LNet: Removed LNI 192.168.203.129@tcp [ 4914.953473] Key type .llcrypt unregistered [ 4914.954920] Key type ._llcrypt unregistered [ 4939.941341] Key type ._llcrypt registered [ 4939.943699] Key type .llcrypt registered [ 4940.028468] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 4940.861627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4941.022888] alg: No test for adler32 (adler32-zlib) [ 4942.056458] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 4942.279157] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 4943.919239] Key type lgssc registered [ 4944.964785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4958.836347] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 10:18:16 (1790777896) [ 4974.890891] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 4985.543991] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 5025.500392] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5025.552918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5026.857147] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5026.900463] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5027.039771] Lustre: lustre-MDT0000: new disk, initializing [ 5027.175550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5027.214589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5031.492667] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 5044.001481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5044.260703] Lustre: 167007:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5044.390703] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5044.399505] Lustre: Skipped 1 previous similar message [ 5044.536956] Lustre: lustre-MDT0001: new disk, initializing [ 5044.744651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5044.803885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5044.836527] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5049.557336] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 5058.956809] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5064.269808] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5071.706652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5072.050210] Lustre: lustre-OST0000: new disk, initializing [ 5072.053654] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5072.061603] Lustre: 168975:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5072.143067] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5078.039947] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 5079.601599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5079.618664] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5079.700611] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5086.733877] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5095.398262] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5095.398541] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5095.410093] LustreError: Skipped 1 previous similar message [ 5095.410991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5095.417243] Lustre: Skipped 1 previous similar message [ 5095.436106] Lustre: Skipped 1 previous similar message [ 5100.513110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5100.517526] Lustre: Skipped 1 previous similar message [ 5100.786549] Lustre: server umount lustre-OST0000 complete [ 5105.634938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5105.652240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5105.660431] Lustre: Skipped 1 previous similar message [ 5105.676014] Lustre: Skipped 1 previous similar message [ 5110.515532] Lustre: server umount lustre-MDT0000 complete [ 5114.027087] LustreError: 166999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790778053 with bad export cookie 6378510085916123162 [ 5114.035395] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5114.040192] LustreError: 166999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5114.362588] Lustre: server umount lustre-MDT0001 complete [ 5123.641474] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 5125.973454] Key type lgssc unregistered [ 5126.299583] LNet: 170879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5126.306555] LNetError: 170879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5126.336361] LNet: Removed LNI 192.168.203.129@tcp [ 5127.142832] Key type .llcrypt unregistered [ 5127.144612] Key type ._llcrypt unregistered [ 5151.586786] Key type ._llcrypt registered [ 5151.589408] Key type .llcrypt registered [ 5151.690265] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 5153.165830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5153.190050] alg: No test for adler32 (adler32-zlib) [ 5154.312136] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 5154.712816] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 5156.407234] Key type lgssc registered [ 5157.604095] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5165.241761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5165.280475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5166.781887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5171.574891] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 5176.294438] LustreError: 172308:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5176.317850] LustreError: 172308:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5180.035229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5180.489086] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5185.153260] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 5192.851992] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5198.830736] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5205.211280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5205.630728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5211.762390] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5211.916638] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 5220.147327] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5227.141318] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5227.168805] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5227.189141] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5227.195434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5227.212293] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5228.201403] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5228.226822] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5228.239617] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5228.247401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5228.260771] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5229.351524] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5229.392717] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5229.408990] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5229.414259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5229.431829] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5230.426256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5231.371759] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5231.383794] Lustre: Skipped 1 previous similar message [ 5231.406231] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5231.422789] Lustre: Skipped 1 previous similar message [ 5231.427087] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5231.433428] Lustre: Skipped 1 previous similar message [ 5231.438112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5231.445837] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5231.454367] Lustre: Skipped 1 previous similar message [ 5232.500342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5233.454798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5234.422583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5235.240581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5236.092209] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5236.098954] Lustre: Skipped 4 previous similar messages [ 5236.120524] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5236.135151] Lustre: Skipped 4 previous similar messages [ 5236.145731] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5236.158263] Lustre: Skipped 4 previous similar messages [ 5236.164127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5236.172620] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5236.180055] Lustre: Skipped 4 previous similar messages [ 5237.219901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5238.087806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5239.042946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5239.975769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5240.802144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5241.703397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5242.756329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5243.623235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5244.464188] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5244.468912] Lustre: Skipped 8 previous similar messages [ 5244.484356] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5244.497179] Lustre: Skipped 8 previous similar messages [ 5244.501121] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5244.505628] Lustre: Skipped 8 previous similar messages [ 5244.509614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5244.516220] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5244.522666] Lustre: Skipped 8 previous similar messages [ 5245.263840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5246.076976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5246.890918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5247.743261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5248.581782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5249.378197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5250.239459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5250.957406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5251.976073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5252.797510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5253.639466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5254.590836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5255.499247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5256.259497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5257.199542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5258.124490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5259.012716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5260.023748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5260.874435] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5260.885763] Lustre: Skipped 18 previous similar messages [ 5260.907611] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5260.932190] Lustre: Skipped 18 previous similar messages [ 5260.935334] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5260.940531] Lustre: Skipped 18 previous similar messages [ 5260.953275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5260.979609] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5260.993505] Lustre: Skipped 18 previous similar messages [ 5261.920978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5262.928706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5263.932533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5264.922644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5265.856104] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5266.840194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5267.781968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5268.750581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5269.739569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5270.651791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5271.627740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5272.538172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5273.519256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5274.377945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5275.137826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5276.120356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5277.104037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5278.001227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5279.008065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5280.053598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5280.930661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5281.886984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5282.902590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5283.888113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5284.894698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5286.203079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5287.250304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5288.272503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5289.229471] LustreError: 172328:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5289.236918] LustreError: 172328:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5289.241981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5290.104276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5291.016723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5291.799479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5292.863397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5293.896635] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5293.904092] Lustre: Skipped 33 previous similar messages [ 5293.920642] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5293.939588] Lustre: Skipped 33 previous similar messages [ 5293.946461] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5293.957317] Lustre: Skipped 33 previous similar messages [ 5293.964020] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5293.989373] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5293.998076] Lustre: Skipped 33 previous similar messages [ 5294.801895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5295.723128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5296.683693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5297.606690] LustreError: 172328:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5297.620036] LustreError: 172328:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5297.624235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5298.561533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5299.688424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5300.614864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5301.565526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5302.663845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5303.589298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5304.543310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5305.326922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5306.272353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5307.202293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5308.243307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5309.292185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5310.305050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5311.303505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5312.251451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5313.179658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5314.283164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5315.425163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5316.399193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5317.554281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5318.624418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5319.555390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5320.509554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5321.370873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5322.454517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5323.359502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5324.318714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5325.178457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5326.205257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5327.095890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5328.136347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5329.062729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5329.943676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5330.928855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5332.053286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5333.026199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5334.102669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5335.084526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5336.189625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5337.139535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5338.082704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5339.121629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5340.059745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5341.030304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5342.261933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5343.265195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5344.299485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5345.360043] LustreError: 172328:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5345.380785] LustreError: 172328:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5345.388429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5346.358224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5347.263823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5348.088608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5349.212356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5350.191060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5351.090935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5352.053494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5353.011829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5353.996885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5354.966843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5355.989612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5357.112917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5358.229506] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5358.234229] Lustre: Skipped 64 previous similar messages [ 5358.259160] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5358.271638] Lustre: Skipped 64 previous similar messages [ 5358.282627] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5358.289871] Lustre: Skipped 64 previous similar messages [ 5358.297946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5358.319800] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5358.328190] Lustre: Skipped 64 previous similar messages [ 5359.326049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5360.514902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5361.511896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5362.656752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5363.737695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5364.702364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5365.842125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5366.840094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5367.898472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5368.918564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5369.814554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5370.704929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5371.583067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5372.634745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5373.566902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5374.565061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5375.495620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5376.525406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5377.605257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5378.695181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5379.761175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5380.759962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5381.560763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5382.498151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5383.428901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5384.364654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5385.412584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5386.180470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5387.104029] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5388.248693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5389.196726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5390.354314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5391.340922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5392.479151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5393.612921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5394.631231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5395.953837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.136267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5398.224357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5399.283632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5400.366308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5401.374702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5402.448295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5403.413985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5404.422154] LustreError: 172328:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5404.451931] LustreError: 172328:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5404.465894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5405.549726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5406.457252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5407.414673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.779141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5410.206979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5411.267757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5412.331607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5413.456876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5414.599132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5415.576653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5416.621849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5417.661689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.646603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5419.625822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5420.715341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5421.795327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5422.832225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5423.804503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5424.996451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5425.942816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5427.044853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5428.167782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5429.144362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5430.240977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5431.315788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5432.409696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5433.851902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5435.211673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5436.270249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5437.549466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5438.655030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5439.733727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5440.679289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5441.773071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5442.914923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5444.025058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5445.006214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5446.152762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5447.276391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5448.279423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5449.259649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5450.036058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5451.051372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5452.003041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5453.053827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5454.133374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5455.154979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5456.110273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5457.186293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5458.189362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5459.070560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5460.005274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5460.965987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.940736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5462.934835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5464.337945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5465.544967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5466.776552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.852800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.036786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5470.246804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5471.495198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5472.667362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.634222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5474.856885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.911479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5476.815156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5477.817874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5478.914879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5479.930234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.114061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.342435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5483.355294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.254226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.221099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5486.161728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5487.339131] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5487.348335] Lustre: Skipped 121 previous similar messages [ 5487.365462] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5487.373540] Lustre: Skipped 121 previous similar messages [ 5487.376954] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5487.391048] Lustre: Skipped 121 previous similar messages [ 5487.396566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5487.414080] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5487.420536] Lustre: Skipped 121 previous similar messages [ 5488.555944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.729447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5490.920453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.089577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.969035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5494.000875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.193334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5496.407530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.813975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.044640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.316128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.502173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5502.756206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5503.838750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.844410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.906162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.246751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.482142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5509.424887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.596262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5511.571419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.701972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.639614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5514.685588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.667625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5516.643033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.642367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.706830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5519.652167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.709589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.732833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5522.593119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.555742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.459229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5525.433943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5526.386880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.399374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.295037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.276466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5530.203642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.190373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.211342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5533.160390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.248878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5535.332625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.475397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5537.531630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.637372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5539.711536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.826649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5541.842268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.006233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.005897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.079846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.111643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.153951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.119029] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.105156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.066243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.257705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.387563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.320253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.354994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.242757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5556.569580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.575565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.597953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.540466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.709698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.676708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.642602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.686159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.817707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5565.826650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.708288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.686280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.585970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.488984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.494163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.332167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.275936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.454377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.368168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.211440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.110862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.049764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.940200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.895868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.821820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.729688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.686929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.605750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.431301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.399883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.313305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5586.349955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.326743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.298848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.226636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.259197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.328710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.385240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.251535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.145530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.939598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.861963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.737477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.475236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.390352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.431673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.374814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.425753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.474304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.392653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.431327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.317293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.273412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.400856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.294157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.311336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.188499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.409883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.342870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.396673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.396683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.329803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.394674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.343545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.203368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.098465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.946154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.946224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.866043] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.756829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.558661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.355738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.266921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.141284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.242546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.098488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.828908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.656762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.330275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.153545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.091079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.024923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.948477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.809660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.652162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.659586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.472894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.392330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.273078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.100577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.842844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.801489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.768236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.642528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.433628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.320401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.162188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.069782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.881456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.755784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.820658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.711325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.643040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.483868] LustreError: 172328:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5652.499159] LustreError: 172328:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5652.513124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.438388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.237971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.121656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.292147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.119649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.076152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.943990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.771267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.595633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.463843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.485046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.505885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.404673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.387451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.253507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.158269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.888558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.671908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.590650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.484845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.206952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.987200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.703681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.435289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.207493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.030847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.911923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.737862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.553507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.509351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.426067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.391940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.463228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.368208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.385789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.378938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.333258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.256214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.224166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.077934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.882605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.652567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.347535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.150672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.945690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.755656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.517081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.346107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.119440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.993176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.915064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.746089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.538596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.292820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.972741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.915965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.784734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.660838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.409668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.288937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.066081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.908769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.784647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.757572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.641791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.433950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.252449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.236265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5712.391210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.315157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.226210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.062873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.977765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.910950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.946644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.799873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.782142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.817124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.617640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.543155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.439676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.417573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.323146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.311753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.106156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.861219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.666478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.644792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.607478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.561271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.747891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.820777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.758142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.979830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.079748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.322313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.434087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5740.793174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.951240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.814665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.697527] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5743.704343] Lustre: Skipped 268 previous similar messages [ 5743.731543] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5743.740972] Lustre: Skipped 268 previous similar messages [ 5743.746722] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5743.755237] Lustre: Skipped 268 previous similar messages [ 5743.758632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.767553] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5743.774269] Lustre: Skipped 268 previous similar messages [ 5744.811315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.925155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.141302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.258706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.569717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5750.391522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.398441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.484434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.741727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.928662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.187184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.250925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.133506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.150661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.206773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.231676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.269752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.389237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.539625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.653403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.785392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.984278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.005239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.869932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.725564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.763341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.712554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.725729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.661516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.706989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.916789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.014609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.033174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.086715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.107530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.996515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.903300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.380927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.445131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.698025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.679873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.761238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.464822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.964949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.522094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.904140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.192792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.611965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.347030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.199932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.649246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.140261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.608926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.517185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.869544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.734029] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.753587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.056733] LustreError: 172328:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5810.077685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.080190] LustreError: 172328:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5811.415170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.461943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.397226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5814.494550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.379601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.191942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.103842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.129087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.118571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.096682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.342330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.273634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.147546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.015926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.918354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.788048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.677761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.573994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.401375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.154781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.910425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.813627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.678329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.332210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.148511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.006208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.849599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5835.810266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.674880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.501541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.234755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.101268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.028147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.862177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.894157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.750231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.655306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5844.602238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.446842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.248672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.089590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.993820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.141220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.177635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.231392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.051918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.044372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.837443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.709525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.414599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.268663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5857.388757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.507665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.659714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.832938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.804803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.720669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.707257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.660821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.521069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.355754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.276953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.163150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.052844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.906319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.759515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.609641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.384221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.305229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.123537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5875.101499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.041961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.075652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.124055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.220344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.096520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5881.225395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5882.388547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.472012] LustreError: 172328:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5883.482845] LustreError: 172328:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5883.485843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.380634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.311291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.411886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.388744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.580583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.824583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.057136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.532826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.953771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.231834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.481530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.571501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.608518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.880684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.924132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5901.922629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.138786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.254496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.210180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.280134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.245302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.274071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.362187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.481520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.407677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.537232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.474776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5914.580331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.632922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.482088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.192301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.077984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.944607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.141841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5921.231658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.589698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.001674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.385419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.689032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5927.852477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5928.679738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5929.624894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5930.567713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.833840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.991715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5934.138964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5935.073686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5936.329485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5937.304821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5938.503339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.698596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.928825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.066770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.993298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5944.003706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5945.359521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.302511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5947.564354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.506665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.582066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5950.431910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5951.980280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.313765] LustreError: 172328:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5953.334107] LustreError: 172328:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5953.350353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5954.563051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5955.507750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5956.538924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5957.489593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5958.492711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5959.513973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5960.517424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5961.566659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5962.413214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5963.521890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5964.557920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5965.715533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5966.776358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.723214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5968.867905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5970.047166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5971.078755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5972.173610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5973.128142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5974.197561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.083886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.958387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.901955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5977.899422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.774812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.636019] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5980.678640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5981.581417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5982.519770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5983.393227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.290944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5985.069498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.026843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.868404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.006857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5989.080060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5990.113712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.107966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.975428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5992.785169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5993.640338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5994.432951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5995.361092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5996.082576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5996.910455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5997.630754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.329303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5999.069793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5999.861411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.527360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6001.312770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6002.068200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6002.832972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6003.634743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6004.444338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6005.243689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.069475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.882593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6007.621889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6008.544354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6009.431612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6010.574007] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6011.560136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6012.548173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6013.508462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.446942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6015.591268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6016.620883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6017.531292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6018.368475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6019.329560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6020.271079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6021.290154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6022.175977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6023.020064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6023.854962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6024.809732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6025.758582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6026.653973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6027.450594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6028.427530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6029.308813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6030.288359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6031.285432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6032.163216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6033.178540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.136375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6035.206718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6036.073468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6036.993353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6037.969923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.967619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6039.835112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6040.835062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6041.728690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6042.725461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6043.513082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6044.541345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6045.317358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6046.253422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6047.232767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6048.073118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6049.027282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6049.914471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6050.791023] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6051.547141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6052.387823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6053.124235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6053.959866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6054.749960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6055.597743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.368756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6057.241568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6058.024478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6058.913682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6059.638874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6060.641831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6061.401103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6062.240901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6062.992722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6063.791488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.527162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6065.400220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6066.221087] LustreError: 172328:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6066.234761] LustreError: 172328:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6066.237525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6067.100841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6068.166915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6068.994777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6069.762649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6070.594598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6071.508319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6072.442375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6073.341795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6074.234817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.217601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6076.177823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6077.136068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6077.996532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6078.853278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6079.686101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6080.645622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6081.459430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6082.359728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6083.267498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6084.146554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6085.030729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6085.835192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6086.748965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6087.687437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6088.606381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6089.501779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6090.464311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6091.573874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6092.519047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6093.457373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6094.476131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6095.697290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6096.800994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6097.836926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6098.866776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6099.882906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6100.934741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6101.876234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6102.951275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6103.826251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6104.737788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6105.752393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6106.705541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6107.747321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6108.652027] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6109.524152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6110.457899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6111.449589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6112.315236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6113.397528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6114.286366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6115.227802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6116.270726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6117.093381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6117.929418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6118.790940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6119.886739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6120.996214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6121.880714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6122.830286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6123.724560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6124.731629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6125.616705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6126.433441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6127.384404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6128.389726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6129.391893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6130.672672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6132.259408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6133.157169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6134.351113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6135.499174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6136.335636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6137.383351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6138.309416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6139.301134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6140.317223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6141.543972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6142.431547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6143.465954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6144.369822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6145.240255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6146.286990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6147.227192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6148.186965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6149.363731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6150.389475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6151.472165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6152.767257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6153.746718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6154.667253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6155.641212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6156.461286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6157.519865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6158.796927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6159.643543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6160.622328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6161.714517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6162.651379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6163.948984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6165.111319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6166.292850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6167.281909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6168.318557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6169.285898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6170.438796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6171.572772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6172.743817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6173.862949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6174.707153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6175.742779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6176.684590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6177.638898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6178.566848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6179.597734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6180.514269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6181.564603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6182.667318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6183.660231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6184.666125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6185.630055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6186.548931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6187.542377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6188.565739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6189.654727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6190.730163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6191.692232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6192.785471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6193.726038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6194.628125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6195.625674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6196.535389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6197.438403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6198.331481] LustreError: 172328:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6198.340983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6198.347722] LustreError: 172328:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6199.463540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6200.333917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6201.253115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6202.119057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6203.017636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6203.911496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6204.965062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6206.001824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6207.029909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6207.957064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6208.742820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6209.529770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6210.412582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6211.348379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6630.036841] LustreError: 174130:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6630.047816] LustreError: 172328:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 7370.729244] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7370.735057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7370.739077] Lustre: Skipped 473 previous similar messages [ 7375.569352] Lustre: server umount lustre-OST0000 complete [ 7380.973139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7380.979440] Lustre: Skipped 2 previous similar messages [ 7386.080820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7386.092917] Lustre: Skipped 1 previous similar message [ 7386.618698] Lustre: server umount lustre-MDT0000 complete [ 7390.191348] LustreError: 184056:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7390.214397] LustreError: 184056:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7391.227902] LustreError: 208203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790780331 with bad export cookie 1317628916228425132 [ 7391.230082] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7391.251665] LustreError: 208203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7391.819909] Lustre: server umount lustre-MDT0001 complete [ 7405.777735] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 7408.721849] Key type lgssc unregistered [ 7409.076805] LNet: 227584:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7409.086298] LNetError: 227584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7409.113590] LNet: Removed LNI 192.168.203.129@tcp [ 7410.219336] Key type .llcrypt unregistered [ 7410.221543] Key type ._llcrypt unregistered [ 7439.030303] Key type ._llcrypt registered [ 7439.032667] Key type .llcrypt registered [ 7439.132068] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 7440.613481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7440.643028] alg: No test for adler32 (adler32-zlib) [ 7441.740296] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 7441.947346] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 7443.615283] Key type lgssc registered [ 7444.804511] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7453.571606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7453.619989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7455.174432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7460.552111] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7464.420173] LustreError: 229017:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7464.453198] LustreError: 229017:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7469.545958] LustreError: 229016:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7469.743583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7470.321735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7475.895696] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7486.083407] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7493.243905] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7500.580555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7500.885377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7507.281252] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7515.608724] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7524.501824] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 7532.580745] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7534.634691] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7534.650335] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 7534.671661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7534.683203] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7534.685527] LustreError: 229036:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 7537.132615] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7537.141695] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7537.163136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7538.246125] Lustre: lustre-OST0000: Not available for connect from 192.168.203.29@tcp (stopping) [ 7540.192357] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7540.199117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7540.247646] Lustre: Skipped 1 previous similar message [ 7543.059450] Lustre: server umount lustre-OST0000 complete [ 7554.767167] LustreError: 229036:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 7578.603626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7579.148184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7581.158120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7581.195053] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7585.887773] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7590.457054] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7590.482538] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7594.789769] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7607.331232] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 8 sec [ 7614.846170] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffffa00e87098000.ost_server_uuid 50 [ 7617.212882] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffffa00e87098000.ost_server_uuid in FULL state after 0 sec [ 7622.626490] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7622.635223] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7622.662485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7623.846656] Lustre: server umount lustre-OST0000 complete [ 7629.283041] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7629.290001] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7629.308684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7634.793024] Lustre: server umount lustre-MDT0000 complete [ 7636.455431] LustreError: 229016:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7636.478634] LustreError: 229016:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7638.520662] LustreError: 229753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790780578 with bad export cookie 13589720865676736257 [ 7638.531379] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7638.545294] LustreError: 229753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7648.999971] Lustre: server umount lustre-MDT0001 complete [ 7662.782240] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 7666.458946] Key type lgssc unregistered [ 7666.882760] LNet: 234280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7666.892101] LNetError: 234280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7666.918252] LNet: Removed LNI 192.168.203.129@tcp [ 7668.143110] Key type .llcrypt unregistered [ 7668.156208] Key type ._llcrypt unregistered [ 7703.364094] Key type ._llcrypt registered [ 7703.365495] Key type .llcrypt registered [ 7703.475516] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 7704.515376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7704.730850] alg: No test for adler32 (adler32-zlib) [ 7705.811720] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 7706.079276] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 7707.794523] Key type lgssc registered [ 7708.879556] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7718.011718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7718.043754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7719.429875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7724.109525] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7729.636807] LustreError: 236025:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7729.656858] LustreError: 236025:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7732.715559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7733.163802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7737.596881] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7740.694740] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7746.814507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7747.133364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7753.125199] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7761.541829] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7761.734687] Lustre: lustre-OST0001: new disk, initializing [ 7761.738671] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7761.745082] Lustre: 238332:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7761.835924] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7764.937709] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 7767.646124] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7768.246966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7768.253887] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7768.413345] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7778.094136] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7786.495374] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7799.267186] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7799.271580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7799.282470] Lustre: Skipped 2 previous similar messages [ 7799.292739] Lustre: Skipped 3 previous similar messages [ 7804.399043] LustreError: 236020:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7804.428701] LustreError: 236020:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7804.634479] Lustre: server umount lustre-MDT0000 complete [ 7808.140657] LustreError: 236005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790780747 with bad export cookie 9894251669167025816 [ 7808.144574] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7808.158250] LustreError: 236005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7808.440729] Lustre: server umount lustre-MDT0001 complete [ 7822.597298] Lustre: server umount lustre-OST0000 complete [ 7825.866393] Lustre: 235282:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790780749/real 1790780749] req@ffff8d0d176e7800 x1877769605700096/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790780765 ref 1 fl Rpc:EXNQr/200/ffffffff rc -5/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7836.369625] Lustre: server umount lustre-OST0001 complete [ 7855.282696] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 7863.700185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7864.051168] LustreError: 241439:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7864.066994] LustreError: 241439:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7864.105599] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7867.983947] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7869.423691] LustreError: 241440:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7873.507072] LustreError: 241439:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7875.258268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7875.739234] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7880.474552] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7883.537775] Lustre: 242580:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7890.419766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7897.764182] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7900.984665] LustreError: 242933:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7907.020982] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7907.192917] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7907.204782] Lustre: Skipped 1 previous similar message [ 7908.231839] Lustre: mylustre-OST0001: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 7908.251341] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49730 to 0x280000401:49760) [ 7912.877037] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7920.406219] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7924.298947] Lustre: 244452:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7933.922846] Lustre: mylustre-MDT0000-lwp-OST0000: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7933.938146] Lustre: Skipped 2 previous similar messages [ 7933.939706] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7933.958383] Lustre: Skipped 3 previous similar messages [ 7939.045523] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7939.049802] Lustre: Skipped 3 previous similar messages [ 7944.166805] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7944.180429] Lustre: Skipped 3 previous similar messages [ 7948.255950] Lustre: mylustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7948.422734] Lustre: server umount mylustre-MDT0000 complete [ 7949.294565] LustreError: 241436:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7949.333730] LustreError: 241436:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 7952.326408] LustreError: 241416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790780892 with bad export cookie 9894251669167028595 [ 7952.337151] LustreError: 241416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7952.339183] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7952.789028] Lustre: server umount mylustre-MDT0001 complete [ 7967.986923] Lustre: server umount mylustre-OST0000 complete [ 7970.591160] Lustre: 235281:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790780894/real 1790780894] req@ffff8d0d1eeadc00 x1877769605761152/t0(0) o400->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790780910 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7970.627718] Lustre: mylustre-MDT0001-lwp-OST0001: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7970.645100] Lustre: Skipped 2 previous similar messages [ 7972.174521] Lustre: server umount mylustre-OST0001 complete [ 7995.553202] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 8005.769628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8006.141726] LustreError: 246760:0:(ldlm_lib.c:1199:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8006.232688] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8010.476234] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8018.198734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8022.712134] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8025.239859] Lustre: 247901:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8031.729420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8037.645849] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8045.843978] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8051.202391] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49730 to 0x280000401:49792) [ 8051.219543] Lustre: tfs-OST0001: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 8051.902364] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8058.900946] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8062.618469] Lustre: 249775:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8069.600309] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8069.605399] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8069.619790] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 8074.544899] Lustre: server umount tfs-MDT0000 complete [ 8076.779270] LustreError: 246761:0:(ldlm_lib.c:1199:target_handle_connect()) tfs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8076.796648] LustreError: 246761:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 8077.872182] LustreError: 246736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790781017 with bad export cookie 9894251669167030485 [ 8077.874667] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8077.883247] LustreError: 246736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8078.141326] Lustre: server umount tfs-MDT0001 complete [ 8092.253840] Lustre: server umount tfs-OST0000 complete [ 8104.576179] Lustre: server umount tfs-OST0001 complete [ 8127.318667] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 8137.403820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8137.853137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8137.869127] Lustre: Skipped 3 previous similar messages [ 8141.960606] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8149.131698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8153.081677] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8155.862209] Lustre: 253167:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8161.952432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8163.311940] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49730 to 0x280000401:49824) [ 8168.302472] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8176.836373] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8182.275903] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 8183.285450] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8191.133608] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8194.708234] Lustre: 255037:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8202.788681] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 11:12:19 (1790781139) [ 8221.153817] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8221.169272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8221.191523] Lustre: Skipped 3 previous similar messages [ 8221.203566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8221.220625] Lustre: Skipped 4 previous similar messages [ 8225.033449] Lustre: server umount lustre-MDT0000 complete [ 8228.327540] LustreError: 253891:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8228.361569] LustreError: 253891:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 8228.481291] LustreError: 252003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790781168 with bad export cookie 9894251669167032151 [ 8228.499599] LustreError: 252003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8228.500213] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8233.446148] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8233.461534] Lustre: Skipped 3 previous similar messages [ 8235.091488] Lustre: server umount lustre-MDT0001 complete [ 8244.971237] Lustre: server umount lustre-OST0000 complete [ 8254.682984] Lustre: server umount lustre-OST0001 complete [ 8261.832462] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 8271.532834] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 8307.562351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8307.880315] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8307.920875] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8308.009546] Lustre: lustre-MDT0000: new disk, initializing [ 8308.096722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 188-180 [ 8308.106362] Lustre: Skipped 3 previous similar messages [ 8308.127956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8312.146107] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8323.162936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8323.248578] Lustre: 258222:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8323.318907] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8323.324748] Lustre: Skipped 1 previous similar message [ 8323.418358] Lustre: lustre-MDT0001: new disk, initializing [ 8323.545907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8323.552615] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8327.796756] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8338.261538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8338.649428] Lustre: lustre-OST0000: new disk, initializing [ 8338.653481] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8338.662896] Lustre: 259806:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8340.154814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8340.186136] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8340.390066] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8346.949746] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8358.151965] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8358.421500] Lustre: lustre-OST0001: new disk, initializing [ 8358.430358] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8358.443650] Lustre: 260679:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8359.900698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8359.930938] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8360.098614] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8365.179157] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8377.235685] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 11:15:15 (1790781315) [ 8394.845363] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 11:15:31 (1790781331) [ 8411.122551] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8411.136969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8411.144553] Lustre: Skipped 2 previous similar messages [ 8411.164050] Lustre: Skipped 7 previous similar messages [ 8417.064290] Lustre: server umount lustre-OST0000 complete [ 8420.836872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8426.962934] Lustre: server umount lustre-MDT0000 complete [ 8430.939869] LustreError: 258212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790781370 with bad export cookie 9894251669167033705 [ 8430.951754] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8430.954848] LustreError: 258212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8437.387790] Lustre: server umount lustre-MDT0001 complete [ 8447.409165] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 8522.412649] Lustre: server umount lustre-OST0001 complete [ 8529.237715] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 8545.512845] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 8584.595594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8585.048967] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8585.096623] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8585.248234] Lustre: lustre-MDT0000: new disk, initializing [ 8585.400925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8585.413420] Lustre: Skipped 3 previous similar messages [ 8585.437033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8590.467686] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8601.821946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8601.910495] Lustre: 265538:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8601.998447] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8602.012348] Lustre: Skipped 1 previous similar message [ 8602.147713] Lustre: lustre-MDT0001: new disk, initializing [ 8602.316385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8602.332838] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8606.920804] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8617.323256] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8623.421762] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8631.765730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8631.988182] Lustre: 267508:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8633.839666] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8633.932076] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8640.908369] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8652.215916] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8664.550299] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8664.552339] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8664.563407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8664.563430] Lustre: Skipped 11 previous similar messages [ 8664.567263] Lustre: Skipped 5 previous similar messages [ 8666.085925] Lustre: server umount lustre-OST0000 complete [ 8674.272733] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8676.649033] Lustre: server umount lustre-MDT0000 complete [ 8679.904956] LustreError: 265550:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8679.921813] LustreError: 265550:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 8680.363921] LustreError: 266342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790781620 with bad export cookie 9894251669167035175 [ 8680.367319] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8680.375902] LustreError: 266342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8680.758886] Lustre: server umount lustre-MDT0001 complete [ 8690.823848] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 8693.532337] Key type lgssc unregistered [ 8693.795758] LNet: 269269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8693.802331] LNetError: 269269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8693.815915] LNet: Removed LNI 192.168.203.129@tcp [ 8694.685197] Key type .llcrypt unregistered [ 8694.688130] Key type ._llcrypt unregistered [ 8721.832292] Key type ._llcrypt registered [ 8721.834424] Key type .llcrypt registered [ 8721.895542] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 8723.001702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8723.150753] alg: No test for adler32 (adler32-zlib) [ 8724.376566] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 8724.643553] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 8726.327496] Key type lgssc registered [ 8727.835685] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8736.660932] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8736.699422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8738.149752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8742.756616] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8747.495653] LustreError: 270748:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8747.540951] LustreError: 270748:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8752.395321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8752.611993] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8752.764299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8757.474786] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8765.686312] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8772.013674] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8779.569572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8780.040276] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8783.106794] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 8786.187751] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8793.783280] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8794.884228] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 8798.117810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8798.120922] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8803.321792] LustreError: 272568:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8803.517585] Lustre: server umount lustre-OST0000 complete [ 8808.417886] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8808.450576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8808.466850] Lustre: Skipped 2 previous similar messages [ 8813.537335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8813.545656] Lustre: Skipped 1 previous similar message [ 8814.500899] Lustre: server umount lustre-MDT0000 complete [ 8818.563493] LustreError: 270728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790781758 with bad export cookie 14938808569192822894 [ 8818.566919] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8818.583781] LustreError: 270728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8819.087478] Lustre: server umount lustre-MDT0001 complete [ 8831.153620] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 8834.162144] Key type lgssc unregistered [ 8834.435151] LNet: 274334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8834.440685] LNetError: 274334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8834.452260] LNet: Removed LNI 192.168.203.129@tcp [ 8835.216860] Key type .llcrypt unregistered [ 8835.218795] Key type ._llcrypt unregistered [ 8854.377660] Key type ._llcrypt registered [ 8854.382311] Key type .llcrypt registered [ 8854.523573] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 8855.617971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8855.709579] alg: No test for adler32 (adler32-zlib) [ 8856.768926] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 8857.093685] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 8858.807251] Key type lgssc registered [ 8860.114718] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8869.654949] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8869.700437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8885.343899] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8890.463355] LustreError: 275543:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.129@tcp: failed processing log, type 4: rc = -110 [ 8917.151216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8924.041942] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8932.143664] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8934.282652] Lustre: server umount lustre-OST0000 complete [ 8952.571918] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 8956.383602] Key type lgssc unregistered [ 8956.718659] LNet: 277174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8956.728051] LNetError: 277174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8956.767534] LNet: Removed LNI 192.168.203.129@tcp [ 8957.879718] Key type .llcrypt unregistered [ 8957.882303] Key type ._llcrypt unregistered [ 9004.354376] Key type ._llcrypt registered [ 9004.356697] Key type .llcrypt registered [ 9004.505990] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 9006.857550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9006.941092] alg: No test for adler32 (adler32-zlib) [ 9008.190128] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 9008.605830] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 9010.440252] Key type lgssc registered [ 9011.841714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9028.326385] loop0: detected capacity change from 0 to 204800000 [ 9028.504802] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9028.521942] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9028.962973] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9030.497804] loop0: detected capacity change from 0 to 204800000 [ 9030.759743] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9030.778177] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9031.271163] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9032.226549] loop0: detected capacity change from 0 to 204800000 [ 9032.439037] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9032.451903] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9032.527684] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9034.053923] loop0: detected capacity change from 0 to 204800000 [ 9034.261701] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9034.277046] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9034.422137] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9035.613377] loop0: detected capacity change from 0 to 204800000 [ 9035.707742] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9041.322356] loop0: detected capacity change from 0 to 204800000 [ 9041.350979] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9046.002586] loop0: detected capacity change from 0 to 204800000 [ 9046.041423] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9050.957937] loop0: detected capacity change from 0 to 204800000 [ 9051.010505] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9055.833708] loop0: detected capacity change from 0 to 204800000 [ 9055.924384] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9055.998444] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9056.103494] Lustre: 280610:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9056.116432] Lustre: 280610:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 9056.128114] Lustre: 280610:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 9056.144561] Lustre: 280610:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9056.163374] Lustre: 280610:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9056.172620] Lustre: 280610:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9056.181339] Lustre: 280610:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 9056.193232] Lustre: 280610:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 9056.199042] Lustre: 280610:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 9056.206180] Lustre: 280610:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 9056.217533] Lustre: 280610:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 9056.224620] Lustre: 280610:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 9056.232526] Lustre: 280610:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 9056.241253] Lustre: 280610:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 9062.678845] Lustre: server umount MGS complete [ 9063.590107] loop0: detected capacity change from 0 to 204800000 [ 9063.726466] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9064.654905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9064.727981] Lustre: lustre-MDT0000: Aborting client recovery [ 9066.127772] loop1: detected capacity change from 0 to 204800000 [ 9066.417196] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9066.643081] Lustre: 281044:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9066.679931] Lustre: 281044:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 9066.724712] Lustre: 281044:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 9066.731901] Lustre: 281044:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9066.744970] Lustre: 281044:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 9066.764449] Lustre: 281044:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9066.783439] Lustre: 281044:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9066.886450] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9067.274188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9067.375099] Lustre: lustre-MDT0001: Aborting client recovery [ 9068.655859] loop2: detected capacity change from 0 to 204800000 [ 9068.904416] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9069.011295] Lustre: 281124:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9069.036393] Lustre: 281124:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9069.063543] Lustre: 281124:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9069.102774] Lustre: 281124:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9069.123280] Lustre: 281124:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9069.143380] Lustre: 281124:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9069.158638] Lustre: 281124:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9069.304855] Lustre: Found index 0 for lustre-OST0000, updating log [ 9069.356350] Lustre: 281141:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9069.438139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9069.474440] Lustre: lustre-OST0000: Aborting recovery [ 9070.848881] loop3: detected capacity change from 0 to 204800000 [ 9070.978048] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9071.244831] Lustre: 281211:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9071.257722] Lustre: 281211:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9071.275312] Lustre: 281211:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9071.289542] Lustre: 281211:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9071.299871] Lustre: 281211:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9071.311744] Lustre: 281211:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9071.320644] Lustre: 281211:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9071.341919] Lustre: Found index 1 for lustre-OST0001, updating log [ 9071.377810] Lustre: 281218:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9071.450855] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9071.475523] Lustre: lustre-OST0001: Aborting recovery [ 9071.501613] LustreError: 280988:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -108 [ 9074.542608] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9074.543545] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9074.578637] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 9074.611877] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 9077.606784] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 9077.608666] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 9081.182513] LustreError: 280955:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 235us: rc = -2 [ 9081.695679] LustreError: 281147:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 5887us: rc = -2 [ 9081.723956] LustreError: 281147:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 20 previous similar messages [ 9098.209352] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9098.211106] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9098.212202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9098.232724] Lustre: Skipped 3 previous similar messages [ 9100.130674] Lustre: server umount lustre-MDT0000 complete [ 9100.864075] LustreError: 280938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790782040 with bad export cookie 1452195897951704584 [ 9100.874273] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9103.328534] LustreError: 280953:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9103.331033] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9103.335490] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9103.335499] Lustre: Skipped 3 previous similar messages [ 9103.347216] LustreError: 280953:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9103.393465] Lustre: Skipped 1 previous similar message [ 9105.378523] LustreError: 281608:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9105.378932] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9105.392028] LustreError: 281608:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9105.416432] Lustre: Skipped 2 previous similar messages [ 9107.292714] Lustre: server umount lustre-MDT0001 complete [ 9114.356138] Lustre: server umount lustre-OST0000 complete [ 9121.637652] Lustre: server umount lustre-OST0001 complete [ 9129.383052] Lustre: DEBUG MARKER: == conf-sanity test 108c: mkfs.lustre --mmp option enables MMP feature ========================================================== 11:27:47 (1790782067) [ 9152.153891] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 9191.986704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9192.220552] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9192.253617] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9192.347523] Lustre: lustre-MDT0000: new disk, initializing [ 9192.437112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9192.484721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9197.056886] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9203.616996] Lustre: server umount lustre-MDT0000 complete [ 9220.197881] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 11:29:18 (1790782158) [ 9250.989992] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 9260.353144] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 9299.197823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9299.504411] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9299.560830] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9299.573511] Lustre: Skipped 1 previous similar message [ 9299.682659] Lustre: lustre-MDT0000: new disk, initializing [ 9299.758816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9299.786261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9306.464830] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9317.615809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9317.688576] Lustre: 288320:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9317.765077] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9317.769522] Lustre: Skipped 1 previous similar message [ 9317.827964] Lustre: lustre-MDT0001: new disk, initializing [ 9317.910277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9317.966534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9317.976793] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9322.326139] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9332.137105] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9337.605129] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9343.873542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9344.079808] Lustre: lustre-OST0000: new disk, initializing [ 9344.083348] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9344.091880] Lustre: 290292:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9350.024816] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9351.758353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9351.777876] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9351.954610] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9358.906229] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9363.942236] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9375.904723] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9383.648214] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9428.969667] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9428.977826] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9428.978569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9428.991171] Lustre: Skipped 1 previous similar message [ 9431.663116] Lustre: server umount lustre-OST0000 complete [ 9439.209613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9439.210908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9439.229541] Lustre: Skipped 1 previous similar message [ 9439.241346] Lustre: Skipped 2 previous similar messages [ 9442.433140] Lustre: server umount lustre-MDT0000 complete [ 9444.322872] LustreError: 288333:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9444.344600] LustreError: 288333:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9445.947429] LustreError: 288314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790782385 with bad export cookie 1452195897951707650 [ 9445.956622] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9445.968976] LustreError: 288314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9446.283778] Lustre: server umount lustre-MDT0001 complete [ 9455.788189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9460.559479] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9465.552056] Lustre: server umount MGS complete [ 9477.372388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9477.871468] LustreError: 294205:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9477.909879] LustreError: 294218:0:(qmt_pool.c:1648:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9477.972468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9477.977360] Lustre: Skipped 1 previous similar message [ 9483.175312] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9483.243339] LustreError: 294206:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9491.791711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9496.921563] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9505.657429] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9513.183910] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9522.041746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9529.612841] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9537.761849] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9567.712672] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9567.718240] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9567.730859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9567.764530] Lustre: Skipped 1 previous similar message [ 9567.794154] Lustre: Skipped 1 previous similar message [ 9571.898306] Lustre: server umount lustre-OST0000 complete [ 9577.953283] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9577.973117] Lustre: Skipped 1 previous similar message [ 9582.681152] Lustre: server umount lustre-MDT0000 complete [ 9583.074108] LustreError: 297042:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9583.085222] LustreError: 297042:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9586.181286] LustreError: 296051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790782526 with bad export cookie 1452195897951709463 [ 9586.185373] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9586.192022] LustreError: 296051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9586.709578] Lustre: server umount lustre-MDT0001 complete [ 9598.870436] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 9601.683967] Key type lgssc unregistered [ 9601.983897] LNet: 298148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9602.003420] LNetError: 298148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9602.032165] LNet: Removed LNI 192.168.203.129@tcp [ 9602.803185] Key type .llcrypt unregistered [ 9602.806705] Key type ._llcrypt unregistered [ 9640.202891] Key type ._llcrypt registered [ 9640.212270] Key type .llcrypt registered [ 9640.356540] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 9656.688205] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 9657.740621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9657.862036] alg: No test for adler32 (adler32-zlib) [ 9659.017457] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [ 9659.303760] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 9660.999304] Key type lgssc registered [ 9662.167112] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9698.824861] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9698.862924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9700.177765] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9700.212690] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9700.295958] Lustre: lustre-MDT0000: new disk, initializing [ 9700.409742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9700.430666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9705.192351] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9716.633544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9716.780144] Lustre: 302121:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9716.890330] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9716.896186] Lustre: Skipped 1 previous similar message [ 9717.024699] Lustre: lustre-MDT0001: new disk, initializing [ 9717.173000] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9717.307824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9717.319711] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9722.301826] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9733.859469] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9740.345571] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9747.381247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9747.629221] Lustre: lustre-OST0000: new disk, initializing [ 9747.637755] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9747.649937] Lustre: 304089:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9747.770454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9753.569362] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9757.757713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9757.772047] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9757.894978] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9763.549169] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9771.143227] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9783.488490] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9787.791339] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9834.982582] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9834.986813] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9834.999458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9835.008103] Lustre: Skipped 1 previous similar message [ 9836.915496] Lustre: server umount lustre-OST0000 complete [ 9845.218307] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9845.223712] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9845.238911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9845.254916] Lustre: Skipped 2 previous similar messages [ 9848.273580] Lustre: server umount lustre-MDT0000 complete [ 9852.318767] LustreError: 305706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790782792 with bad export cookie 11366326691577366909 [ 9852.323114] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9852.333071] LustreError: 305706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9852.739942] Lustre: server umount lustre-MDT0001 complete [ 9861.906823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9866.235035] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9871.112386] Lustre: server umount MGS complete [ 9882.768289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9883.256266] LustreError: 307909:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9883.290212] LustreError: 307909:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9883.323855] LustreError: 307922:0:(qmt_pool.c:1648:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9883.402525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9887.981115] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9888.737446] LustreError: 307910:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9893.861485] LustreError: 307909:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9896.839377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9897.182224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9901.629209] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9909.536222] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9915.606526] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9924.795856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9925.367369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9932.358618] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9940.236500] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9969.634588] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9969.645195] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9969.661794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9969.666835] Lustre: Skipped 1 previous similar message [ 9972.714821] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9972.718861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9972.719594] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9972.758349] Lustre: Skipped 1 previous similar message [ 9975.774774] Lustre: server umount lustre-OST0000 complete [ 9982.945644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9982.966258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9982.986831] Lustre: Skipped 1 previous similar message [ 9993.184954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9993.196109] Lustre: Skipped 4 previous similar messages [ 9994.208202] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9994.495538] Lustre: server umount lustre-MDT0000 complete [ 9998.305633] LustreError: 307909:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9998.332205] LustreError: 307909:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9998.963877] LustreError: 309753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790782938 with bad export cookie 11366326691577368736 [ 9998.991571] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9998.997354] LustreError: 309753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9999.490333] Lustre: server umount lustre-MDT0001 complete [10013.868147] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [10017.382770] Key type lgssc unregistered [10017.724609] LNet: 311858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10017.733193] LNetError: 311858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10017.759230] LNet: Removed LNI 192.168.203.129@tcp [10018.942456] Key type .llcrypt unregistered [10018.945952] Key type ._llcrypt unregistered [10050.159225] Key type ._llcrypt registered [10050.166258] Key type .llcrypt registered [10050.360111] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [10066.041078] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [10067.078438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10067.163523] alg: No test for adler32 (adler32-zlib) [10068.454258] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [10068.742475] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [10070.463204] Key type lgssc registered [10071.829286] Lustre: Echo OBD driver; http://www.lustre.org/ [10101.332453] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [10117.222856] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10117.259546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10118.498461] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10118.528382] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10118.625935] Lustre: lustre-MDT0000: new disk, initializing [10118.706263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10118.746884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10124.456447] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [10136.183214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10136.480963] Lustre: lustre-OST0000: new disk, initializing [10136.487899] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10136.494463] Lustre: Skipped 1 previous similar message [10136.505621] Lustre: 315928:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10136.613307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10138.240300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10138.257406] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10138.309058] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10143.424748] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11117.026678] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11117.046221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11119.167571] Lustre: server umount lustre-MDT0000 complete [11122.961271] LustreError: 315116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790784062 with bad export cookie 1843657448009270393 [11122.965868] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11122.977843] LustreError: 315116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11123.201770] Lustre: server umount lustre-OST0000 complete [11149.861359] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [11160.446802] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [11202.710335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11202.964756] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11203.008711] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11203.145803] Lustre: lustre-MDT0000: new disk, initializing [11203.212515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11203.232147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11207.811668] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11218.676514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11218.879656] Lustre: 320295:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11218.965742] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11218.972848] Lustre: Skipped 1 previous similar message [11219.027525] Lustre: lustre-MDT0001: new disk, initializing [11219.099309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11219.125926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11219.132941] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11224.404754] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11234.651123] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11240.547570] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11247.577563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11247.897775] Lustre: lustre-OST0000: new disk, initializing [11247.900731] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11247.911949] Lustre: 322261:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11248.018592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11254.409715] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11257.434103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11257.461033] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11257.672469] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11264.117173] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11269.153396] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11269.343672] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11273.907674] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11274.328788] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11278.310606] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11278.319685] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11278.330358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11278.372918] Lustre: Skipped 1 previous similar message [11283.433874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11283.449111] Lustre: Skipped 2 previous similar messages [11288.549187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11288.556227] Lustre: Skipped 1 previous similar message [11291.103135] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11291.289565] Lustre: server umount lustre-OST0000 complete [11296.235909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11296.247139] LustreError: Skipped 1 previous similar message [11296.256609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11296.285262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11298.785189] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11302.346393] Lustre: server umount lustre-MDT0000 complete [11307.507024] LustreError: 320285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790784247 with bad export cookie 1843657448012211219 [11307.516400] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11307.518111] LustreError: 320285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11308.033989] Lustre: server umount lustre-MDT0001 complete [11313.258510] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [11314.785446] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 12:04:12 (1790784252) [11321.975511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11322.476921] LustreError: 324576:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11322.503596] LustreError: 324576:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11322.610815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11327.972671] LustreError: 324577:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11328.321577] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11333.106408] LustreError: 324576:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11338.132499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11338.213250] LustreError: 324576:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11338.596644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11342.714986] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11350.847616] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11357.046450] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11364.691051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11365.144278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11371.256339] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11379.510874] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11388.038241] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11388.209254] Lustre: lustre-OST0001: new disk, initializing [11388.213149] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11388.220474] Lustre: 327414:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11388.305362] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11393.739765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11393.760039] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11393.824009] LustreError: 327674:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11394.856934] LustreError: 327674:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11394.867567] LustreError: 327674:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [11395.937555] LustreError: 327681:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11395.953623] LustreError: 327681:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [11395.954816] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11397.999765] LustreError: 327674:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11398.024254] LustreError: 327674:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [11402.150117] LustreError: 327681:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11402.164960] LustreError: 327681:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [11406.936639] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11407.171096] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11410.416557] LustreError: 327674:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11410.425318] LustreError: 327674:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 15 previous similar messages [11413.833441] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffffa00e87dae800.ost_server_uuid 50 [11415.565639] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffffa00e87dae800.ost_server_uuid in FULL state after 0 sec [11421.586529] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa00e87dae800.ost_server_uuid 50 [11423.310625] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffffa00e87dae800.ost_server_uuid in FULL state after 0 sec [11424.811893] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11440.096846] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11440.117425] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11440.126990] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11440.135645] Lustre: Skipped 1 previous similar message [11440.154255] Lustre: Skipped 3 previous similar messages [11445.332861] Lustre: server umount lustre-OST0001 complete [11447.818087] LustreError: 326395:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11454.433089] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11454.438074] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11458.589411] Lustre: server umount lustre-OST0000 complete [11465.699653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11465.713757] Lustre: Skipped 7 previous similar messages [11469.498584] Lustre: server umount lustre-MDT0000 complete [11470.816862] LustreError: 324573:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11470.838264] LustreError: 324573:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [11472.977439] LustreError: 324557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790784412 with bad export cookie 1843657448012212094 [11472.985459] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11472.996077] LustreError: 324557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11473.458318] Lustre: server umount lustre-MDT0001 complete [11485.286587] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [11488.167205] Key type lgssc unregistered [11488.449581] LNet: 330103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11488.459417] LNetError: 330103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11488.482602] LNet: Removed LNI 192.168.203.129@tcp [11489.424916] Key type .llcrypt unregistered [11489.426861] Key type ._llcrypt unregistered [11515.691693] Key type ._llcrypt registered [11515.697788] Key type .llcrypt registered [11515.839471] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [11516.898258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11517.069329] alg: No test for adler32 (adler32-zlib) [11518.083437] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [11518.356486] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [11520.031255] Key type lgssc registered [11521.304141] Lustre: Echo OBD driver; http://www.lustre.org/ [11531.681558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11531.719549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11533.536308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11539.886988] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11544.034266] LustreError: 331541:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11544.075522] LustreError: 331541:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11549.177354] LustreError: 331540:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11549.339489] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11554.283258] LustreError: 331541:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11556.845808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11557.284044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11563.611967] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11574.728758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11575.056688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11579.176506] LustreError: 333222:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11579.195500] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [11581.995518] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11587.584502] LustreError: 333223:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11587.616819] LustreError: 333223:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11590.425793] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11597.822044] LustreError: 333444:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11597.874983] LustreError: 333444:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11598.366724] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11598.725614] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11603.973320] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [11607.368174] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11616.565788] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11623.495917] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [11623.710502] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11747.306028] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11747.313655] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11747.328958] LustreError: Skipped 1 previous similar message [11747.329665] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11747.338888] Lustre: Skipped 1 previous similar message [11747.383559] Lustre: Skipped 1 previous similar message [11749.199549] Lustre: server umount lustre-OST0001 complete [11752.420199] LustreError: 333223:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11752.444444] LustreError: 333223:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11757.543584] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11757.549508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11757.557841] Lustre: Skipped 1 previous similar message [11757.565117] Lustre: Skipped 1 previous similar message [11762.681234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11762.694661] Lustre: Skipped 1 previous similar message [11767.777036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11770.335242] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11770.710724] Lustre: server umount lustre-OST0000 complete [11777.504422] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11777.528792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11777.570546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11777.578662] Lustre: Skipped 1 previous similar message [11788.259604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11788.262780] Lustre: Skipped 5 previous similar messages [11789.279164] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11789.679698] Lustre: server umount lustre-MDT0000 complete [11793.378600] LustreError: 331540:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11793.407786] LustreError: 331540:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [11794.656629] LustreError: 331522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790784734 with bad export cookie 12718945559147737164 [11794.658415] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11794.675569] LustreError: 331522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11795.256515] Lustre: server umount lustre-MDT0001 complete [11807.810786] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [11810.656574] Key type lgssc unregistered [11811.043284] LNet: 336487:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11811.049222] LNetError: 336487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11811.091237] LNet: Removed LNI 192.168.203.129@tcp [11812.298811] Key type .llcrypt unregistered [11812.304380] Key type ._llcrypt unregistered [11861.888846] Key type ._llcrypt registered [11861.893563] Key type .llcrypt registered [11862.025096] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [11862.983799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11863.042464] alg: No test for adler32 (adler32-zlib) [11864.605113] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [11864.828647] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [11866.543200] Key type lgssc registered [11867.980529] Lustre: Echo OBD driver; http://www.lustre.org/ [11878.348042] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11878.383484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11879.875486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11884.508969] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11890.149980] LustreError: 338796:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11890.165785] LustreError: 338796:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11893.490888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11893.929518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11898.317845] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11901.524356] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11908.199422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11908.683208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11914.767565] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11918.947547] LustreError: 340283:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11923.365228] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11923.606460] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11924.677667] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [11924.693552] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [11930.001938] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11939.878231] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11945.874532] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11960.293444] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11960.312430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11960.325819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11965.013557] Lustre: server umount lustre-MDT0000 complete [11965.924220] LustreError: 338796:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11969.602689] LustreError: 338776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790784909 with bad export cookie 18204937103613697257 [11969.609694] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11969.612822] LustreError: 338776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11970.042968] Lustre: server umount lustre-MDT0001 complete [11983.493827] Lustre: server umount lustre-OST0000 complete [11986.336345] Lustre: 338051:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790784910/real 1790784910] req@ffff8d0e30c8e680 x1877773966701696/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790784926 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11986.373493] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11986.410500] Lustre: Skipped 3 previous similar messages [11988.170093] Lustre: server umount lustre-OST0001 complete [12005.075439] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [12017.636596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12018.176253] LustreError: 344059:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12018.208867] LustreError: 344059:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12018.267212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12023.267150] LustreError: 344060:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12023.566682] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [12033.140900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12033.432722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12038.189233] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [12041.017434] Lustre: 345202:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [12048.451057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12055.248684] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [12059.060672] LustreError: 345557:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12059.098118] LustreError: 345557:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12064.189084] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [12064.369235] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12064.616107] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12064.622614] Lustre: Skipped 1 previous similar message [12069.894939] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [12071.379799] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [12079.722908] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12083.369932] Lustre: 347072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [12089.824640] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12089.835700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12089.862172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12089.868377] Lustre: Skipped 3 previous similar messages [12094.171833] Lustre: server umount lustre-MDT0000 complete [12095.471226] LustreError: 344059:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12095.496120] LustreError: 344059:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [12099.096838] LustreError: 344041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790785038 with bad export cookie 18204937103613698958 [12099.119283] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12099.121766] LustreError: 344041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12099.690714] Lustre: server umount lustre-MDT0001 complete [12114.585656] Lustre: server umount lustre-OST0000 complete [12116.959815] Lustre: 338048:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790785040/real 1790785040] req@ffff8d0e03721500 x1877773966749184/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790785056 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12117.028552] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12117.039107] Lustre: Skipped 3 previous similar messages [12118.631917] Lustre: server umount lustre-OST0001 complete [12123.225920] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [12125.140630] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [12126.815519] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 12:17:44 (1790785064) [12152.157941] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [12160.405560] loop0: detected capacity change from 0 to 1099511627776 [12161.065262] SGI XFS with ACLs, security attributes, quota, no debug enabled [12161.077583] *** VALIDATE xfs *** [12161.109627] XFS (loop0): Mounting V5 Filesystem [12161.165468] XFS (loop0): Ending clean mount [12162.732353] loop1: detected capacity change from 0 to 18691697672192 [12286.399123] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [12297.080430] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [12336.912918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12337.169270] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12337.220129] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12337.315586] Lustre: lustre-MDT0000: new disk, initializing [12337.439711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12337.456533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12342.151080] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [12352.554773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12352.672445] Lustre: 352211:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12352.723385] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12352.727289] Lustre: Skipped 1 previous similar message [12352.796519] Lustre: lustre-MDT0001: new disk, initializing [12352.903242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12352.918585] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12357.745413] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [12368.256582] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12374.634605] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12381.800540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12382.074609] Lustre: lustre-OST0000: new disk, initializing [12382.080639] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12382.089345] Lustre: 354179:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12384.108639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12384.146839] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12384.281923] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12388.860879] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [12398.018064] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12402.568432] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12402.781924] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12407.698986] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12408.025936] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12414.947236] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12414.960179] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12414.985208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12414.989549] Lustre: Skipped 4 previous similar messages [12416.407180] Lustre: server umount lustre-OST0000 complete [12424.673704] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12424.681613] LustreError: Skipped 1 previous similar message [12424.687437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12424.705554] Lustre: Skipped 1 previous similar message [12424.711546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12426.824375] Lustre: server umount lustre-MDT0000 complete [12430.305913] LustreError: 352217:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12430.333696] LustreError: 352217:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12430.827173] LustreError: 352202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790785370 with bad export cookie 18204937103613700512 [12430.830270] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12430.834677] LustreError: 352202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12431.229647] Lustre: server umount lustre-MDT0001 complete [12440.016199] XFS (loop0): Unmounting Filesystem [12444.626212] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 12:23:02 (1790785382) [12451.532200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12452.077515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12452.088394] Lustre: Skipped 2 previous similar messages [12457.306405] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [12466.797792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12472.111678] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [12480.831615] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12488.366376] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12495.884891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12502.845601] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [12511.450335] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12527.072624] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12527.091161] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12527.111519] Lustre: Skipped 1 previous similar message [12527.127699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12527.135121] Lustre: Skipped 2 previous similar messages [12531.107863] Lustre: server umount lustre-OST0000 complete [12538.347603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12538.357583] Lustre: Skipped 3 previous similar messages [12541.932325] Lustre: server umount lustre-MDT0000 complete [12543.461418] LustreError: 356736:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12543.473205] LustreError: 356736:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [12546.922626] LustreError: 356721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790785486 with bad export cookie 18204937103613701394 [12546.926846] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12546.934473] LustreError: 356721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12547.264791] Lustre: server umount lustre-MDT0001 complete [12561.363077] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [12564.244340] Key type lgssc unregistered [12564.596555] LNet: 360369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12564.612110] LNetError: 360369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12565.670145] LNet: Removed LNI 192.168.203.129@tcp [12566.619188] Key type .llcrypt unregistered [12566.621897] Key type ._llcrypt unregistered [12596.738598] Key type ._llcrypt registered [12596.741275] Key type .llcrypt registered [12596.905448] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [12598.294855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12598.335090] alg: No test for adler32 (adler32-zlib) [12599.505156] Lustre: Lustre: Build Version: 2.17.59_4_ge4c1f18 [12599.753690] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [12601.455188] Key type lgssc registered [12602.847786] Lustre: Echo OBD driver; http://www.lustre.org/ [12610.475588] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12610.515995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12612.081628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12616.933639] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [12621.283734] LustreError: 361952:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12621.306297] LustreError: 361952:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12626.328647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12626.406500] LustreError: 361953:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12626.862821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12632.618338] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [12644.380885] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12651.592842] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12660.201516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12660.848887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12668.134732] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [12678.895314] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12692.964341] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12692.967402] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12692.981163] Lustre: Skipped 1 previous similar message [12692.993689] Lustre: Skipped 1 previous similar message [12695.587430] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.29@tcp (stopping) [12698.084215] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12698.098495] Lustre: Skipped 1 previous similar message [12700.678649] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.29@tcp (stopping) [12705.247126] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12705.475465] Lustre: server umount lustre-MDT0001 complete [12705.812588] LustreError: 364445:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12708.323766] LustreError: 363059:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12708.350243] LustreError: 363059:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12713.445200] LustreError: 361952:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12713.460721] LustreError: 361952:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12716.262271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12716.402896] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12716.419730] Lustre: Found index 1 for lustre-MDT0001, updating log [12716.489574] Lustre: 361942:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12716.883947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12722.168211] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12722.856954] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [12724.212881] LustreError: 365476:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12724.219982] LustreError: 365476:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12724.229648] LustreError: 361945:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12724.331136] LustreError: 365481:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.129@tcp: cfg command failed: rc = -17 [12724.348312] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [12724.348312] [12732.401089] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12732.407595] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12732.419809] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12732.426161] Lustre: Skipped 2 previous similar messages [12737.506491] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12738.753560] Lustre: server umount lustre-MDT0001 complete [12741.634185] LustreError: 364445:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12741.652291] LustreError: 364445:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12749.838230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12750.028428] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12750.041665] Lustre: Found index 1 for lustre-MDT0001, updating log [12750.126232] Lustre: 361940:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12750.494915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12755.168856] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [12755.958715] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12757.995279] LustreError: 366618:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12758.010128] LustreError: 366618:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12758.025919] LustreError: 361945:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12758.047355] LustreError: 361945:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12758.120623] LustreError: 366633:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [12758.129215] LustreError: 366633:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [12758.135963] LustreError: 366633:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.129@tcp: cfg command failed: rc = -17 [12758.148164] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.203.129@tcp [12758.148164] [13062.186136] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.29@tcp (stopping) [13062.196248] Lustre: Skipped 3 previous similar messages [13063.136878] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13063.152967] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13067.931257] Lustre: server umount lustre-MDT0001 complete [13071.330973] LustreError: 361948:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13071.351764] LustreError: 361948:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [13079.474660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13079.533910] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13079.541107] Lustre: Found index 1 for lustre-MDT0001, updating log [13079.569532] Lustre: 361942:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13079.840055] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13084.440309] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [13085.188479] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13085.195647] LustreError: 374978:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13085.205515] LustreError: 374978:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13085.215026] LustreError: 361945:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13085.224335] LustreError: 361945:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [13085.272090] LustreError: 374986:0:(lod_lov.c:208:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [13085.284256] LustreError: 374986:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.129@tcp: cfg command failed: rc = -114 [13085.291298] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [13085.291298] [13407.713404] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13407.726221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13407.751243] Lustre: Skipped 1 previous similar message [13407.765761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13407.779866] Lustre: Skipped 4 previous similar messages [13413.005733] Lustre: server umount lustre-MDT0000 complete [13415.907678] LustreError: 361949:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13415.939025] LustreError: 361949:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [13416.787485] LustreError: 361934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790786356 with bad export cookie 9330295754238259674 [13416.789121] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13416.796611] LustreError: 361934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13421.025939] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13421.044611] Lustre: Skipped 2 previous similar messages [13431.263089] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13431.478435] Lustre: server umount lustre-MDT0001 complete [13435.037865] Lustre: server umount lustre-OST0000 complete [13444.138460] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [13453.538376] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [13488.988953] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 12:40:26 (1790786426) [13495.095197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13495.349382] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13495.374233] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13495.441295] Lustre: lustre-MDT0000: new disk, initializing [13495.493765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13495.521612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13500.028954] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [13509.499678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13509.645779] Lustre: 385833:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13509.738304] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13509.745035] Lustre: Skipped 1 previous similar message [13509.806496] Lustre: lustre-MDT0001: new disk, initializing [13509.870152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13509.904233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13509.918358] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13514.440727] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [13523.509697] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13529.299805] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13535.450446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13535.720295] Lustre: lustre-OST0000: new disk, initializing [13535.727239] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13535.742688] Lustre: 387804:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13535.838593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13541.579175] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [13545.559636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13545.574483] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13545.681760] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13550.692204] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13559.784109] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13559.792432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13559.814533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13559.818757] Lustre: Skipped 6 previous similar messages [13560.353930] Lustre: server umount lustre-MDT0000 complete [13561.316141] LustreError: 385844:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13561.334328] LustreError: 385844:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [13563.759086] LustreError: 386629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790786503 with bad export cookie 9330295754238261627 [13563.769203] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13563.772956] LustreError: 386629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13563.798318] LustreError: 386641:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [13564.127066] Lustre: server umount lustre-MDT0001 complete [13577.890618] Lustre: server umount lustre-OST0000 complete [13606.778323] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [13614.935791] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [13642.325995] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13337 sec ========= 12:43:00 (1790786580) [13643.762627] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:43:01 (1790786581) === [13646.692682] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:43:04 (1790786584) === [13686.191326] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [13688.686651] Key type lgssc unregistered [13688.960100] LNet: 394164:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13688.979263] LNetError: 394164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13688.996523] LNet: Removed LNI 192.168.203.129@tcp [13689.741024] Key type .llcrypt unregistered [13689.745716] Key type ._llcrypt unregistered