[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 685841081 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002234] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.007679] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009010] pid_max: default: 32768 minimum: 301 [ 0.010132] LSM: Security Framework initializing [ 0.011046] Yama: becoming mindful. [ 0.012000] SELinux: Initializing. [ 0.012068] *** VALIDATE selinux *** [ 0.021633] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026368] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028083] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029140] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030113] *** VALIDATE tmpfs *** [ 0.031607] *** VALIDATE proc *** [ 0.032291] *** VALIDATE cgroup *** [ 0.033007] *** VALIDATE cgroup2 *** [ 0.034327] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035167] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037033] Spectre V2 : User space: Vulnerable [ 0.038013] Speculative Store Bypass: Vulnerable [ 0.041929] debug: unmapping init [mem 0xffffffff8fc59000-0xffffffff8fc60fff] [ 0.045000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045802] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046030] ... version: 2 [ 0.047015] ... bit width: 48 [ 0.048015] ... generic registers: 4 [ 0.049019] ... value mask: 0000ffffffffffff [ 0.050012] ... max period: 00007fffffffffff [ 0.051018] ... fixed-purpose events: 3 [ 0.052016] ... event mask: 000000070000000f [ 0.053448] rcu: Hierarchical SRCU implementation. [ 0.055736] smp: Bringing up secondary CPUs ... [ 0.056686] x86: Booting SMP configuration: [ 0.057026] .... node #0, CPUs: #1 #2 #3 [ 0.066207] smp: Brought up 1 node, 4 CPUs [ 0.068014] smpboot: Max logical packages: 1 [ 0.069012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.134021] node 0 deferred pages initialised in 63ms [ 0.139221] devtmpfs: initialized [ 0.141964] x86/mm: Memory block size: 128MB [ 0.152010] gcov: version magic: 0x41383552 [ 0.159915] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.162091] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.165770] pinctrl core: initialized pinctrl subsystem [ 0.169528] [ 0.170011] ************************************************************* [ 0.172012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.175014] ** ** [ 0.177013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.179011] ** ** [ 0.185013] ** This means that this kernel is built to expose internal ** [ 0.187009] ** IOMMU data structures, which may compromise security on ** [ 0.191013] ** your system. ** [ 0.194013] ** ** [ 0.197012] ** If you see this message and you are not debugging the ** [ 0.199011] ** kernel, report this immediately to your vendor! ** [ 0.202016] ** ** [ 0.204010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.207013] ************************************************************* [ 0.209546] NET: Registered protocol family 16 [ 0.212869] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.215109] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.217049] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.220270] cpuidle: using governor menu [ 0.222028] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.223561] PCI: Using configuration type 1 for base access [ 0.225143] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.235334] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.236022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.239142] cryptd: max_cpu_qlen set to 1000 [ 0.240208] ACPI: Added _OSI(Module Device) [ 0.241000] ACPI: Added _OSI(Processor Device) [ 0.242012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.244009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.250015] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.256688] ACPI: Interpreter enabled [ 0.258050] ACPI: PM: (supports S0 S3 S4 S5) [ 0.260009] ACPI: Using IOAPIC for interrupt routing [ 0.262103] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.265505] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.277390] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.280029] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.282013] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.286077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.291580] acpiphp: Slot [2] registered [ 0.293154] acpiphp: Slot [5] registered [ 0.294214] acpiphp: Slot [6] registered [ 0.296142] acpiphp: Slot [7] registered [ 0.297078] acpiphp: Slot [8] registered [ 0.299079] acpiphp: Slot [9] registered [ 0.300105] acpiphp: Slot [10] registered [ 0.302088] acpiphp: Slot [3] registered [ 0.303096] acpiphp: Slot [4] registered [ 0.304076] acpiphp: Slot [11] registered [ 0.306089] acpiphp: Slot [12] registered [ 0.307069] acpiphp: Slot [13] registered [ 0.309127] acpiphp: Slot [14] registered [ 0.310087] acpiphp: Slot [15] registered [ 0.312085] acpiphp: Slot [16] registered [ 0.313140] acpiphp: Slot [17] registered [ 0.315079] acpiphp: Slot [18] registered [ 0.316183] acpiphp: Slot [19] registered [ 0.318082] acpiphp: Slot [20] registered [ 0.320173] acpiphp: Slot [21] registered [ 0.321105] acpiphp: Slot [22] registered [ 0.323082] acpiphp: Slot [23] registered [ 0.325091] acpiphp: Slot [24] registered [ 0.326123] acpiphp: Slot [25] registered [ 0.327068] acpiphp: Slot [26] registered [ 0.329127] acpiphp: Slot [27] registered [ 0.330212] acpiphp: Slot [28] registered [ 0.332078] acpiphp: Slot [29] registered [ 0.334105] acpiphp: Slot [30] registered [ 0.335101] acpiphp: Slot [31] registered [ 0.337051] PCI host bridge to bus 0000:00 [ 0.339013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.341016] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.343090] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.346020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.349021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.351021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.353257] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.357107] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.360466] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.370953] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.377664] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.380016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.382019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.385024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.388508] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.391799] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.394038] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.397846] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.404016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.420019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.427018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.433534] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.444028] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.454025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.476031] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.501643] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.523017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.547019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.575030] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.593526] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.606019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.618022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.652017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.673595] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.682000] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.701018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.729017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.748000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.755011] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.762015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.784023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.871025] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.883017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.892000] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.914015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.929226] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.933566] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.936504] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.938513] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.941233] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.947024] iommu: Default domain type: Passthrough [ 0.949378] SCSI subsystem initialized [ 0.951143] ACPI: bus type USB registered [ 0.953132] usbcore: registered new interface driver usbfs [ 0.955112] usbcore: registered new interface driver hub [ 0.958204] usbcore: registered new device driver usb [ 0.960158] pps_core: LinuxPPS API ver. 1 registered [ 0.963011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.968061] PTP clock support registered [ 0.970650] EDAC MC: Ver: 3.0.0 [ 0.974100] PCI: Using ACPI for IRQ routing [ 0.977899] NetLabel: Initializing [ 0.979009] NetLabel: domain hash size = 128 [ 0.982013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.986325] NetLabel: unlabeled traffic allowed by default [ 0.990305] vgaarb: loaded [ 0.992308] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.995016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.002219] clocksource: Switched to clocksource kvm-clock [ 1.156128] VFS: Disk quotas dquot_6.6.0 [ 1.160253] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.163841] *** VALIDATE ramfs *** [ 1.165701] *** VALIDATE hugetlbfs *** [ 1.170988] pnp: PnP ACPI init [ 1.174952] pnp: PnP ACPI: found 6 devices [ 1.198968] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.204278] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.206802] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.210125] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.212840] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.215557] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.218795] NET: Registered protocol family 2 [ 1.220896] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.225050] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.228186] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.234386] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.237673] TCP: Hash tables configured (established 65536 bind 65536) [ 1.239696] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.243834] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.248059] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.251152] NET: Registered protocol family 1 [ 1.254489] RPC: Registered named UNIX socket transport module. [ 1.256783] RPC: Registered udp transport module. [ 1.258369] RPC: Registered tcp transport module. [ 1.260027] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.262275] NET: Registered protocol family 44 [ 1.263600] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.265487] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.267320] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.269482] PCI: CLS 0 bytes, default 64 [ 1.271085] Unpacking initramfs... [ 2.872553] debug: unmapping init [mem 0xffff93e4fcc54000-0xffff93e4fffbffff] [ 2.877655] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.880833] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.884768] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.853708] Initialise system trusted keyrings [ 4.860601] Key type blacklist registered [ 4.876599] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.918518] zbud: loaded [ 4.944803] *** VALIDATE nfs *** [ 4.946441] *** VALIDATE nfs4 *** [ 4.952445] pstore: using deflate compression [ 4.963511] Platform Keyring initialized [ 5.127610] NET: Registered protocol family 38 [ 5.130553] Key type asymmetric registered [ 5.132583] Asymmetric key parser 'x509' registered [ 5.135982] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.142195] io scheduler mq-deadline registered [ 5.145630] io scheduler kyber registered [ 5.148349] io scheduler bfq registered [ 5.151850] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.157723] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.170654] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.180560] ACPI: Power Button [PWRF] [ 5.191161] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.204387] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.233993] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.263548] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.318995] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.349473] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.385123] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.401547] Non-volatile memory driver v1.3 [ 5.404965] Linux agpgart interface v0.103 [ 5.462889] virtio_blk virtio1: [vda] 145792 512-byte logical blocks (74.6 MB/71.2 MiB) [ 5.468408] vda: detected capacity change from 0 to 74645504 [ 5.508545] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.514102] vdb: detected capacity change from 0 to 1073741824 [ 5.574505] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.589445] vdc: detected capacity change from 0 to 2621440000 [ 5.635393] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.640818] vdd: detected capacity change from 0 to 2621440000 [ 5.688240] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.691666] vde: detected capacity change from 0 to 4294967296 [ 5.731631] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.735283] vdf: detected capacity change from 0 to 4294967296 [ 5.750115] libphy: Fixed MDIO Bus: probed [ 5.787922] usbcore: registered new interface driver usbserial_generic [ 5.795226] usbserial: USB Serial support registered for generic [ 5.799922] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.811665] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.817233] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.824908] mousedev: PS/2 mouse device common for all mice [ 5.832550] rtc_cmos 00:05: RTC can wake from S4 [ 5.839711] rtc_cmos 00:05: registered as rtc0 [ 5.841743] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.846046] intel_pstate: CPU model not supported [ 5.851973] hid: raw HID events driver (C) Jiri Kosina [ 5.856908] usbcore: registered new interface driver usbhid [ 5.861688] usbhid: USB HID core driver [ 5.863882] drop_monitor: Initializing network drop monitor service [ 5.866288] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.878836] Initializing XFRM netlink socket [ 5.881871] NET: Registered protocol family 10 [ 5.889552] Segment Routing with IPv6 [ 5.891070] NET: Registered protocol family 17 [ 5.895854] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.899711] mpls_gso: MPLS GSO support [ 5.915522] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.926820] RAS: Correctable Errors collector initialized. [ 5.932175] AVX version of gcm_enc/dec engaged. [ 5.937686] AES CTR mode by8 optimization enabled [ 6.074565] sched_clock: Marking stable (6074488603, 0)->(7408851858, -1334363255) [ 6.079899] registered taskstats version 1 [ 6.082994] Loading compiled-in X.509 certificates [ 6.087215] zswap: loaded using pool lzo/zbud [ 6.172774] Key type big_key registered [ 6.217056] Key type encrypted registered [ 6.219610] ima: No TPM chip found, activating TPM-bypass! [ 6.225281] ima: Allocated hash algorithm: sha1 [ 6.228216] ima: No architecture policies found [ 6.232857] evm: Initialising EVM extended attributes: [ 6.249503] evm: security.selinux [ 6.256170] evm: security.ima [ 6.262833] evm: security.capability [ 6.265844] evm: HMAC attrs: 0x1 [ 6.301724] rtc_cmos 00:05: setting system clock to 2026-07-14 15:42:35 UTC (1784043755) [ 6.322438] debug: unmapping init [mem 0xffffffff90c03000-0xffffffff90dfffff] [ 6.329226] debug: unmapping init [mem 0xffffffff8f982000-0xffffffff8fc58fff] [ 6.338120] Write protecting the kernel read-only data: 28672k [ 6.353815] debug: unmapping init [mem 0xffffffff8e003000-0xffffffff8e1fffff] [ 6.373064] debug: unmapping init [mem 0xffffffff8e914000-0xffffffff8e9fffff] [ 6.520254] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.552880] systemd[1]: Detected virtualization kvm. [ 6.555029] systemd[1]: Detected architecture x86-64. [ 6.562915] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.655902] systemd[1]: No hostname configured. [ 6.659263] systemd[1]: Set hostname to . [ 6.663918] random: systemd: uninitialized urandom read (16 bytes read) [ 6.672747] systemd[1]: Initializing machine ID from random generator. [ 6.846593] random: ln: uninitialized urandom read (6 bytes read) [ 7.101125] random: systemd: uninitialized urandom read (16 bytes read) [ 7.103314] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 7.116919] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ 7.161650] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.557248] device-mapper: uevent: version 1.0.3 [ 8.559787] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 10.643667] virtio_net virtio0 ens2: renamed from eth0 [ 10.831780] random: fast init done [ 10.996151] scsi host0: ata_piix [ 11.077524] scsi host1: ata_piix [ 11.079028] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 11.082678] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.812505] random: crng init done [ 15.817037] random: 7 urandom warning(s) missed due to ratelimiting [ 16.867479] dracut-initqueue[606]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 19.082805] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.434316] printk: systemd: 26 output lines suppressed due to ratelimiting [ 24.294292] SELinux: Disabled at runtime. [ 24.707548] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 24.728789] systemd[1]: Detected virtualization kvm. [ 24.732208] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 28.271180] systemd[1]: initrd-switch-root.service: Succeeded. [ 28.282855] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 28.297846] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 28.309961] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 28.334745] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 28.357095] systemd[1]: Starting Journal Service... Starting Journal Service... [ 29.158772] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 29.726323] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 29.740848] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ 29.751602] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 29.762127] systemd[1]: Starting udev Coldplug all Devices... Starting udev Coldplug all Devices... [ 29.781702] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ 29.803067] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ 29.848710] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ 29.859656] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ 29.873189] systemd[1]: Stopped target Initrd Root File System. [ OK ] Stopped target Initrd Root File System. [ 29.892373] systemd[1]: Stopped target Initrd File Systems. [ OK ] Stopped target Initrd File Systems. [ 29.940897] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... [ OK ] Reached target Paths. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 31.902029] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 34.243807] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 34.305398] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 34.999220] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 35.234726] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…only root support (10s / no limit) [** ] A start job is running for Configur…only root support (11s / no limit) [*** ] A start job is running for Configur…only root support (11s / no limit) [ *** ] A start job is running for Configur…only root support (12s / no limit) [ *** ] A start job is running for Configur…only root support (12s / no limit) [ ***] A start job is running for Configur…only root support (13s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit) [ *] A start job is running for Configur…only root support (14s / no limit) [ **] A start job is running for Configur…only root support (14s / no limit) [ ***] A start job is running for Configur…only root support (15s / no limit)[ 44.092846] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (16s / no limit) [ *** ] A start job is running for Configur…only root support (16s / no limit) [*** ] A start job is running for Configur…only root support (17s / no limit) [** ] A start job is running for Configur…only root support (17s / no limit)[ 46.111869] NFS: Registering the id_resolver key type [ 46.131668] Key type id_resolver registered [ 46.141974] Key type id_legacy registered [* ] A start job is running for Configur…only root support (18s / no limit) [** ] A start job is running for Configur…only root support (18s / no limit) [*** ] A start job is running for Configur…only root support (19s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg407-server login: [ 86.344187] hrtimer: interrupt took 2298132 ns [ 106.833832] libcfs: loading out-of-tree module taints kernel. [ 106.926836] Key type ._llcrypt registered [ 106.943837] Key type .llcrypt registered [ 107.139082] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 126.390552] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 127.855455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 127.874860] alg: No test for adler32 (adler32-zlib) [ 129.389345] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 130.340830] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 132.103201] Key type lgssc registered [ 133.858282] Lustre: Echo OBD driver; http://www.lustre.org/ [ 148.101583] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 198.772232] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 213.506329] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 213.586672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 214.980219] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 215.039107] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 215.187487] Lustre: lustre-MDT0000: new disk, initializing [ 215.302276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 215.331429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 221.571728] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 237.275508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 237.471652] Lustre: 6525:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 237.538221] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 237.547392] Lustre: Skipped 1 previous similar message [ 237.728311] Lustre: lustre-MDT0001: new disk, initializing [ 237.809593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 237.850199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 237.874924] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 242.602733] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 247.782434] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 257.263513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 257.512519] Lustre: lustre-OST0000: new disk, initializing [ 257.516354] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 257.524110] Lustre: 8465:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 257.593193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 263.202220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 263.207603] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 263.304452] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 263.379939] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 277.271273] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 277.434625] Lustre: lustre-OST0001: new disk, initializing [ 277.438429] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 277.442697] Lustre: 9536:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 277.496462] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 283.569243] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 285.712256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 285.720407] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 285.782939] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 295.697989] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 302.129763] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 308.990140] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing check_logdir /tmp/testlogs/ [ 314.367911] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing yml_node [ 319.361843] Lustre: DEBUG MARKER: Client: 2.17.54.162 [ 321.800853] Lustre: DEBUG MARKER: MDS: 2.17.54.162 [ 324.335755] Lustre: DEBUG MARKER: OSS: 2.17.54.162 [ 325.688555] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Jul 14 11:47:53 EDT 2026 [ 343.602694] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 345.532093] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 362.464064] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 362.466874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 362.484803] Lustre: Skipped 1 previous similar message [ 362.490907] Lustre: Skipped 2 previous similar messages [ 367.547462] Lustre: server umount lustre-MDT0000 complete [ 371.247960] LustreError: 7476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784044120 with bad export cookie 4712220156608035093 [ 371.260056] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 371.267301] LustreError: 7476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 372.704770] LustreError: 11209:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 372.706201] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 372.728720] LustreError: 11209:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 372.753563] Lustre: Skipped 2 previous similar messages [ 372.759085] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 372.761749] Lustre: Skipped 2 previous similar messages [ 376.736429] LustreError: 11209:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 376.736714] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 376.769521] LustreError: 11209:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 376.793709] Lustre: Skipped 1 previous similar message [ 377.956237] Lustre: server umount lustre-MDT0001 complete [ 382.210889] Lustre: server umount lustre-OST0000 complete [ 386.159631] Lustre: server umount lustre-OST0001 complete [ 394.234384] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 403.381837] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 441.567123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 441.777309] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 441.797911] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 441.850756] Lustre: lustre-MDT0000: new disk, initializing [ 441.910156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 441.925571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 445.612322] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 454.764959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 454.921774] Lustre: 15093:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 454.977823] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 454.981506] Lustre: Skipped 1 previous similar message [ 455.083568] Lustre: lustre-MDT0001: new disk, initializing [ 455.180325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 455.215905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 455.222781] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 459.012135] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 468.472122] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 474.361285] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 482.259863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 482.593136] Lustre: lustre-OST0000: new disk, initializing [ 482.597691] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 482.603643] Lustre: 17061:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 482.710884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 484.006606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 484.057379] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 484.275556] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 490.232126] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 499.656214] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 505.628390] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 505.860428] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 510.715520] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 510.909658] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 515.040584] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 515.041444] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 515.051860] LustreError: Skipped 1 previous similar message [ 515.059313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 515.073235] Lustre: Skipped 2 previous similar messages [ 515.099282] Lustre: Skipped 1 previous similar message [ 520.172877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 520.187991] Lustre: Skipped 1 previous similar message [ 527.327102] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 527.529220] Lustre: server umount lustre-OST0000 complete [ 531.939776] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 531.951192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 531.979661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 531.983057] Lustre: Skipped 2 previous similar messages [ 537.665355] Lustre: server umount lustre-MDT0000 complete [ 540.645661] LustreError: 15103:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 540.678164] LustreError: 15103:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 541.621243] LustreError: 15085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784044290 with bad export cookie 4712220156608036626 [ 541.635984] LustreError: 15085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 541.636072] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 541.921581] Lustre: server umount lustre-MDT0001 complete [ 547.112183] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 11:51:34 (1784044294) [ 552.997366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 553.337982] LustreError: 19327:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 553.421603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 557.623762] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 558.568185] LustreError: 19328:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 564.570974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 569.152797] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 577.003290] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 582.932972] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 589.743174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 595.728793] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 603.993582] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 607.552463] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 615.905074] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 615.925328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 615.941590] Lustre: Skipped 1 previous similar message [ 615.944920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 615.958940] Lustre: Skipped 2 previous similar messages [ 620.455250] Lustre: server umount lustre-MDT0000 complete [ 623.996139] LustreError: 19308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784044373 with bad export cookie 4712220156608037508 [ 623.998848] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 624.019935] LustreError: 19308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 624.096705] LustreError: 19322:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 624.101049] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 624.128921] LustreError: 19322:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 624.140095] Lustre: Skipped 2 previous similar messages [ 624.306090] Lustre: server umount lustre-MDT0001 complete [ 630.391282] Lustre: server umount lustre-OST0000 complete [ 650.822796] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 661.400437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 661.928789] LustreError: 23705:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 661.994554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 661.999438] Lustre: Skipped 2 previous similar messages [ 666.772676] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 675.686405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 680.592516] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 683.470526] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 690.497643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 697.345844] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 706.324821] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 706.420872] Lustre: lustre-OST0001: new disk, initializing [ 706.424801] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 706.430051] Lustre: 26021:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 708.324640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 708.334776] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 708.518151] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 714.291934] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 723.581419] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 727.086430] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 742.886349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 742.897211] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 742.913864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 742.916786] Lustre: Skipped 4 previous similar messages [ 746.084987] Lustre: server umount lustre-MDT0000 complete [ 749.276386] LustreError: 26020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784044498 with bad export cookie 4712220156608038355 [ 749.285068] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 749.286526] LustreError: 26020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 749.537741] LustreError: 23699:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 749.567446] LustreError: 23699:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 749.690678] Lustre: server umount lustre-MDT0001 complete [ 755.471546] Lustre: server umount lustre-OST0000 complete [ 758.890185] Lustre: server umount lustre-OST0001 complete [ 774.482466] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 783.690471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 788.091368] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 796.682012] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 797.087811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 797.092450] Lustre: Skipped 4 previous similar messages [ 801.453176] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 805.245109] Lustre: 30187:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 812.655234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 819.780205] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 822.188452] LustreError: 30541:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 822.210708] LustreError: 30541:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 828.630178] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 836.809576] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 844.259359] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 848.132251] Lustre: 32068:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 855.528356] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 855.530848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 855.545436] Lustre: Skipped 8 previous similar messages [ 855.561093] Lustre: Skipped 7 previous similar messages [ 861.300415] Lustre: server umount lustre-MDT0000 complete [ 865.022791] LustreError: 29789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784044614 with bad export cookie 4712220156608039650 [ 865.030443] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 865.039470] LustreError: 29789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 865.517433] Lustre: server umount lustre-MDT0001 complete [ 879.689339] Lustre: server umount lustre-OST0000 complete [ 881.121365] Lustre: 3658:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784044614/real 1784044614] req@ffff93e57b9e1c00 x1870705594761728/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784044630 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 883.631364] Lustre: server umount lustre-OST0001 complete [ 892.034314] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 11:57:19 (1784044639) [ 920.736610] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 930.792088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 935.377707] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 945.237624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 950.975261] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 953.525118] Lustre: 36021:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 960.902317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 968.177334] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 970.604048] LustreError: 36375:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 970.626476] LustreError: 36375:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 976.569879] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 983.827423] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 991.927282] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 996.043301] Lustre: 37901:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1001.952434] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1001.957743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1001.963124] Lustre: Skipped 1 previous similar message [ 1001.966533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1001.973034] Lustre: Skipped 4 previous similar messages [ 1007.935475] Lustre: server umount lustre-MDT0000 complete [ 1012.801449] LustreError: 37213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784044761 with bad export cookie 4712220156608040756 [ 1012.806502] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1012.813870] LustreError: 37213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1013.301206] Lustre: server umount lustre-MDT0001 complete [ 1027.157349] Lustre: server umount lustre-OST0000 complete [ 1029.599601] Lustre: 3661:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784044762/real 1784044762] req@ffff93e443658e00 x1870705594811008/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784044778 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1032.253365] Lustre: server umount lustre-OST0001 complete [ 1041.183399] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 11:59:48 (1784044788) [ 1057.959862] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 1069.009899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1069.441141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1069.450514] Lustre: Skipped 6 previous similar messages [ 1073.629456] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1082.251914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1088.436548] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1091.666985] Lustre: 41194:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1098.630804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1105.311594] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1114.372391] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1121.625934] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1128.644310] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1132.605724] Lustre: 43082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1133.814274] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1133.819516] Lustre: Skipped 1 previous similar message [ 1139.171380] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1139.175447] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1139.183968] Lustre: Skipped 4 previous similar messages [ 1143.775386] Lustre: server umount lustre-MDT0000 complete [ 1148.499843] LustreError: 40026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784044897 with bad export cookie 4712220156608041855 [ 1148.504255] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1148.510426] LustreError: 40026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1154.830405] Lustre: server umount lustre-MDT0001 complete [ 1162.768652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1172.963301] LustreError: 3657:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff93e444d10e00 x1870705594865024/t0(0) o250->MGC192.168.204.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1179.114054] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1183.734596] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1188.792252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1194.313745] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1194.492173] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1194.500855] Lustre: Skipped 1 previous similar message [ 1201.166645] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1202.751341] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1208.855504] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1210.645387] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1220.069876] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1222.671749] Lustre: server umount lustre-MDT0000 complete [ 1225.858148] LustreError: 40025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784044975 with bad export cookie 4712220156608043045 [ 1225.864081] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1225.872573] LustreError: 40025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1226.137575] Lustre: server umount lustre-MDT0001 complete [ 1240.649609] Lustre: server umount lustre-OST0000 complete [ 1254.615871] Lustre: server umount lustre-OST0001 complete [ 1264.156392] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 12:03:31 (1784045011) [ 1282.354759] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 1292.691627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1293.140350] LustreError: 47693:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1293.154899] LustreError: 47693:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 34 previous similar messages [ 1297.269405] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1305.188814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1310.835602] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1315.914956] Lustre: 48843:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1325.691150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1333.771247] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1341.877148] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1349.035958] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1356.837856] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1362.408628] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1376.490228] Lustre: server umount lustre-MDT0000 complete [ 1377.761236] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1380.346418] LustreError: 47675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784045129 with bad export cookie 4712220156608043836 [ 1380.355445] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1380.377180] LustreError: 47675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1380.908191] Lustre: server umount lustre-MDT0001 complete [ 1395.151330] Lustre: server umount lustre-OST0000 complete [ 1399.007153] Lustre: 3659:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784045132/real 1784045132] req@ffff93e443447b80 x1870705594956800/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784045148 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1399.083146] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1399.118504] Lustre: Skipped 13 previous similar messages [ 1399.669137] Lustre: server umount lustre-OST0001 complete [ 1410.404643] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 12:05:57 (1784045157) [ 1413.289212] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1415.177699] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 12:06:02 (1784045162) [ 1426.237995] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 1435.620674] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 1457.400914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1457.700342] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1457.832161] Lustre: lustre-MDT0000: new disk, initializing [ 1457.936527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1462.119965] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1471.187322] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1477.986496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1478.256916] Lustre: lustre-OST0000: new disk, initializing [ 1478.259606] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1478.262563] Lustre: Skipped 1 previous similar message [ 1478.266090] Lustre: 54918:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1480.077805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1480.089898] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1480.123742] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1484.265346] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1493.744064] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1495.874293] Lustre: DEBUG MARKER: create test files [ 1531.362821] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1531.374752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1531.381142] Lustre: Skipped 19 previous similar messages [ 1535.604673] Lustre: server umount lustre-OST0000 complete [ 1539.666560] Lustre: server umount lustre-MDT0000 complete [ 1550.590921] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 1553.048468] Key type lgssc unregistered [ 1553.318559] LNet: 56546:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1553.344165] LNetError: 56546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1554.410715] LNet: Removed LNI 192.168.204.107@tcp [ 1555.281508] Key type .llcrypt unregistered [ 1555.285156] Key type ._llcrypt unregistered [ 1586.324589] Key type ._llcrypt registered [ 1586.327584] Key type .llcrypt registered [ 1586.421512] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 1587.212725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1587.369102] alg: No test for adler32 (adler32-zlib) [ 1588.456320] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 1588.634454] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 1590.287362] Key type lgssc registered [ 1591.310272] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1598.044669] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1598.061186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1599.511613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1604.021843] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1610.611100] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1616.931604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1617.230354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1622.986416] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1624.438177] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1624.553711] Lustre: 59278:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1624.560087] Lustre: 59278:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1624.564650] Lustre: 59278:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1624.574703] Lustre: 59278:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1624.579970] Lustre: 59278:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1624.585827] Lustre: 59278:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1630.525462] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1636.771127] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1642.635165] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1655.264978] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1655.274143] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1655.291739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1656.981829] Lustre: server umount lustre-OST0000 complete [ 1661.482700] Lustre: server umount lustre-MDT0000 complete [ 1673.753751] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 1677.215836] Key type lgssc unregistered [ 1677.582983] LNet: 60887:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1677.594473] LNetError: 60887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1677.605367] LNet: Removed LNI 192.168.204.107@tcp [ 1678.862173] Key type .llcrypt unregistered [ 1678.864797] Key type ._llcrypt unregistered [ 1709.357230] Key type ._llcrypt registered [ 1709.359182] Key type .llcrypt registered [ 1709.461193] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 1710.700914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1710.777325] alg: No test for adler32 (adler32-zlib) [ 1711.859311] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 1712.058327] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 1713.687434] Key type lgssc registered [ 1714.935476] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1723.005323] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1723.058493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1724.692151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1728.893058] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1736.570937] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1743.575440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1744.052813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1749.167060] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1749.251980] Lustre: 63726:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1749.259992] Lustre: 63726:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1749.268614] Lustre: 63726:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1749.277242] Lustre: 63726:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1749.282495] Lustre: 63726:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1749.288564] Lustre: 63726:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1751.545631] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1759.358382] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1764.563618] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1805.797060] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1805.804567] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1805.820343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1808.621303] Lustre: server umount lustre-OST0000 complete [ 1812.497347] Lustre: server umount lustre-MDT0000 complete [ 1822.644204] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 1824.940823] Key type lgssc unregistered [ 1825.195817] LNet: 65315:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1825.209042] LNetError: 65315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1825.221763] LNet: Removed LNI 192.168.204.107@tcp [ 1825.917466] Key type .llcrypt unregistered [ 1825.919218] Key type ._llcrypt unregistered [ 1841.793790] Key type ._llcrypt registered [ 1841.799554] Key type .llcrypt registered [ 1841.898153] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 1856.674534] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 1857.270751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1857.292078] alg: No test for adler32 (adler32-zlib) [ 1858.488726] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 1858.807867] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 1860.511243] Key type lgssc registered [ 1861.662477] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1895.417656] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1895.437423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1896.738357] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1896.780129] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1896.902417] Lustre: lustre-MDT0000: new disk, initializing [ 1897.007150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1897.042635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1901.332531] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1911.402828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1911.590382] Lustre: 68466:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1911.640836] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1911.644869] Lustre: Skipped 1 previous similar message [ 1911.710457] Lustre: lustre-MDT0001: new disk, initializing [ 1911.819578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1911.852780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1911.868739] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1916.169498] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1927.180886] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1934.439452] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1943.350064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1943.820182] Lustre: lustre-OST0000: new disk, initializing [ 1943.826234] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1943.843469] Lustre: 70434:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1943.939484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1951.336513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1951.358989] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1951.481541] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1951.778820] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1961.363514] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1966.938760] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1967.190723] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1972.396368] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1972.664112] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1977.313581] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1977.320512] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1977.324687] LustreError: Skipped 1 previous similar message [ 1977.327512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1977.339634] Lustre: Skipped 1 previous similar message [ 1977.351866] Lustre: Skipped 1 previous similar message [ 1980.932928] Lustre: server umount lustre-OST0000 complete [ 1987.564529] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1987.572470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1987.592605] Lustre: Skipped 1 previous similar message [ 1987.608694] Lustre: Skipped 1 previous similar message [ 1991.902365] Lustre: server umount lustre-MDT0000 complete [ 1996.275388] LustreError: 68459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784045745 with bad export cookie 2865685391915153124 [ 1996.281360] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1996.290346] LustreError: 68459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1996.846728] Lustre: server umount lustre-MDT0001 complete [ 2006.766117] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 12:15:54 (1784045754) [ 2018.246877] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 2065.203166] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 2076.833662] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 2116.499841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2116.921887] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2116.963639] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2117.065147] Lustre: lustre-MDT0000: new disk, initializing [ 2117.199264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2117.237105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2122.377815] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2133.203459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2133.291472] Lustre: 76458:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2133.339373] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2133.345188] Lustre: Skipped 1 previous similar message [ 2133.430345] Lustre: lustre-MDT0001: new disk, initializing [ 2133.541634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2133.594687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2133.606138] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2137.867929] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2147.705706] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2154.005933] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2161.755201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2162.049821] Lustre: lustre-OST0000: new disk, initializing [ 2162.054742] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2162.068864] Lustre: 78425:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2162.210368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2167.353280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2167.374265] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2167.509028] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2169.005479] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2178.038907] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2182.189208] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2182.393198] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2186.356476] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2186.500398] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2193.382231] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2193.382716] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2193.389876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2193.396501] Lustre: Skipped 1 previous similar message [ 2193.412968] LustreError: Skipped 1 previous similar message [ 2194.807804] Lustre: server umount lustre-OST0000 complete [ 2200.035686] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2200.047606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2200.068260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2200.073161] Lustre: Skipped 1 previous similar message [ 2204.996550] Lustre: server umount lustre-MDT0000 complete [ 2208.248719] LustreError: 76448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784045957 with bad export cookie 2865685391915154006 [ 2208.251690] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2208.260447] LustreError: 76448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2208.619144] Lustre: server umount lustre-MDT0001 complete [ 2217.210536] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 12:19:24 (1784045964) [ 2223.561205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2224.117358] LustreError: 80838:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2224.137526] LustreError: 80838:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2224.184542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2227.686261] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2229.223132] LustreError: 80839:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2234.347166] LustreError: 80838:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2235.545304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2241.608883] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2249.976409] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2256.470487] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2263.867045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2264.325667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2264.334837] Lustre: Skipped 1 previous similar message [ 2270.685658] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2278.094987] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2280.754619] Lustre: 83508:0:(genops.c:1795:obd_export_evict_by_uuid()) MGS: evicting c577c9d3-4039-4777-a185-22630fa04e67 at adminstrative request [ 2281.712527] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2281.726876] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2291.180904] Lustre: MGS: Client c577c9d3-4039-4777-a185-22630fa04e67 (at 0@lo) reconnecting [ 2291.200132] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x27c4f5ae4f20fdc8 to 0x27c4f5ae4f210068 [ 2291.238207] Lustre: MGC192.168.204.107@tcp: Connection restored to 0@lo (at 0@lo) [ 2317.602867] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2317.798405] Lustre: lustre-OST0001: new disk, initializing [ 2317.802039] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2317.806937] Lustre: 83902:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2324.110595] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2327.747340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2327.758027] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2327.903861] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2334.078431] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2338.273280] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2338.290828] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2338.296165] LustreError: Skipped 1 previous similar message [ 2338.296844] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2338.297887] Lustre: Skipped 2 previous similar messages [ 2338.325743] Lustre: Skipped 2 previous similar messages [ 2342.686719] Lustre: server umount lustre-OST0001 complete [ 2343.413653] LustreError: 82659:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2343.428079] LustreError: 82659:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2346.473432] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2346.479469] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2346.500851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2346.504501] Lustre: Skipped 1 previous similar message [ 2348.527830] LustreError: 82659:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2348.534256] LustreError: 82659:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2352.398230] Lustre: server umount lustre-OST0000 complete [ 2358.755660] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2358.759165] LustreError: Skipped 1 previous similar message [ 2362.390833] Lustre: server umount lustre-MDT0000 complete [ 2363.872169] LustreError: 80838:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2363.886741] LustreError: 80838:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2366.371046] LustreError: 80820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784046115 with bad export cookie 2865685391915155560 [ 2366.379525] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2366.387227] LustreError: 80820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2366.727383] Lustre: server umount lustre-MDT0001 complete [ 2376.073500] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 12:22:03 (1784046123) [ 2377.907153] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2379.860375] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 12:22:07 (1784046127) [ 2381.695861] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2383.676460] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 12:22:11 (1784046131) [ 2385.510983] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2387.460737] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 12:22:14 (1784046134) [ 2389.016120] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2390.890715] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 12:22:18 (1784046138) [ 2392.814025] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2399.155643] loop: module loaded [ 2399.185231] loop0: detected capacity change from 0 to 4294967296 [ 2399.242313] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2400.932729] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2404.387178] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2407.554854] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2433.677231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2433.708890] LustreError: 88028:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2433.717031] LustreError: 88028:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2433.724606] LustreError: 88028:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2433.732990] LustreError: 88028:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2452.851245] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 2463.501271] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 2504.366199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2504.708125] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2504.746566] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2504.843774] Lustre: lustre-MDT0000: new disk, initializing [ 2504.953117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2504.962779] Lustre: Skipped 1 previous similar message [ 2504.979180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2510.004127] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2521.552033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2521.720202] Lustre: 90898:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2522.029202] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2527.545982] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2537.767077] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2543.631874] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2550.360701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2550.602052] Lustre: 92867:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2557.846542] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2558.036729] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2558.212631] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2570.354540] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2575.553435] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2575.744910] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2581.764867] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2582.040089] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2589.159559] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2589.159854] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2589.171871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2589.179521] Lustre: Skipped 4 previous similar messages [ 2589.204106] Lustre: Skipped 5 previous similar messages [ 2590.752257] Lustre: server umount lustre-OST0000 complete [ 2600.796587] Lustre: server umount lustre-MDT0000 complete [ 2604.519162] LustreError: 90909:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2604.547529] LustreError: 90909:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2605.273499] LustreError: 90888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784046354 with bad export cookie 2865685391915156169 [ 2605.282305] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2605.285613] LustreError: 90888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2605.740872] Lustre: server umount lustre-MDT0001 complete [ 2615.453280] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 12:26:02 (1784046362) [ 2623.342942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2629.675919] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2639.333490] LustreError: 95279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2639.380443] LustreError: 95279:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2640.500694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2641.052200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-60 [ 2641.070709] Lustre: Skipped 3 previous similar messages [ 2646.567301] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2655.532153] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2663.045391] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2671.359813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2679.309423] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2689.517813] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2696.508613] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2696.994563] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2703.690739] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2704.222466] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2712.362585] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2712.530081] Lustre: lustre-OST0001: new disk, initializing [ 2712.533551] Lustre: Skipped 2 previous similar messages [ 2712.537885] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2712.541119] Lustre: Skipped 3 previous similar messages [ 2712.546207] Lustre: 98655:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2719.398611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2719.410918] Lustre: Skipped 2 previous similar messages [ 2719.418583] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2719.520119] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2720.256340] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2730.628714] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2735.670701] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2735.851312] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2740.254252] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2740.496593] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2759.779624] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2789.425984] Lustre: Failing over lustre-MDT0000 [ 2789.902597] Lustre: server umount lustre-MDT0000 complete [ 2791.396154] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2791.403345] LustreError: 96010:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2791.415275] Lustre: Skipped 4 previous similar messages [ 2791.442375] LustreError: 96010:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2807.778369] Lustre: 66690:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784046540/real 1784046540] req@00000000f2301fd8 x1870707406685568/t0(0) o400->MGC192.168.204.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784046556 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2807.804046] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2816.044280] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2816.048530] LDISKFS-fs (dm-0): recovery complete [ 2816.066318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2818.024417] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x27c4f5ae4f21cadc [ 2818.035416] Lustre: MGC192.168.204.107@tcp: Connection restored to 0@lo (at 0@lo) [ 2818.571595] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2819.999550] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2820.000016] Lustre: 95274:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2823.649338] Lustre: 95279:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2823.665380] Lustre: 95279:0:(ldlm_lib.c:2084:extend_recovery_timer()) Skipped 1 previous similar message [ 2823.693879] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2823.710129] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2823.794547] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2824.112159] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2824.456406] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2824.470238] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 1 previous similar message [ 2824.801272] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2824.805273] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 1 previous similar message [ 2824.813892] Lustre: 101508:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2824.823285] Lustre: 101508:0:(ldlm_lib.c:2084:extend_recovery_timer()) Skipped 7 previous similar messages [ 2825.489858] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2825.499987] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 2 previous similar messages [ 2825.823114] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2825.832179] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 2 previous similar messages [ 2827.115282] Lustre: 101508:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2827.133158] Lustre: 101508:0:(ldlm_lib.c:2084:extend_recovery_timer()) Skipped 6 previous similar messages [ 2827.790516] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2827.808716] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 6 previous similar messages [ 2828.138469] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2828.152642] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 6 previous similar messages [ 2831.344914] Lustre: 101508:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2831.350043] Lustre: 101508:0:(ldlm_lib.c:2084:extend_recovery_timer()) Skipped 12 previous similar messages [ 2831.992762] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2832.008346] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 12 previous similar messages [ 2832.335166] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2832.346212] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 12 previous similar messages [ 2839.624229] Lustre: 101508:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2839.660933] Lustre: 101508:0:(ldlm_lib.c:2084:extend_recovery_timer()) Skipped 25 previous similar messages [ 2840.296229] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2840.307378] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 25 previous similar messages [ 2840.631748] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2840.637086] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 25 previous similar messages [ 2855.821895] Lustre: 101508:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2855.840157] Lustre: 101508:0:(ldlm_lib.c:2084:extend_recovery_timer()) Skipped 50 previous similar messages [ 2856.481642] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2856.494807] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 50 previous similar messages [ 2856.815166] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2856.828914] LustreError: 101508:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 50 previous similar messages [ 2879.023152] Lustre: 101508:0:(ldlm_lib.c:2393:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2879.033922] Lustre: 101508:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2879.044456] Lustre: 101508:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client aa70f3e3-eb79-48bc-8c6b-61720d9208fa@192.168.204.7@tcp [ 2879.061387] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2879.071373] Lustre: 101508:0:(ldlm_lib.c:1913:abort_req_replay_queue()) @@@ aborted: req@000000005b9f138e x1870707391794816/t0(8589934770) o35->aa70f3e3-eb79-48bc-8c6b-61720d9208fa@192.168.204.7@tcp:513/0 lens 392/0 e 0 to 0 dl 1784046638 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2879.114100] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2879.130564] Lustre: lustre-MDT0000: Denying connection for new client aa70f3e3-eb79-48bc-8c6b-61720d9208fa (at 192.168.204.7@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 2879.145620] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2879.196709] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2879.206268] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2879.218170] Lustre: Skipped 2 previous similar messages [ 2879.266984] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:45 to 0x2c0000401:545) [ 2879.280967] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:78 to 0x280000401:577) [ 2895.333618] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2895.350123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2895.364336] Lustre: Skipped 1 previous similar message [ 2895.364667] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2895.364670] LustreError: Skipped 2 previous similar messages [ 2895.395916] Lustre: Skipped 4 previous similar messages [ 2897.500699] Lustre: server umount lustre-OST0000 complete [ 2907.772517] Lustre: server umount lustre-OST0001 complete [ 2917.577885] Lustre: server umount lustre-MDT0000 complete [ 2920.932056] LustreError: 96010:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2920.962192] LustreError: 96010:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 43 previous similar messages [ 2921.153801] LustreError: 95259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784046670 with bad export cookie 2865685391915207388 [ 2921.165506] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2921.168610] LustreError: 95259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2921.459400] Lustre: server umount lustre-MDT0001 complete [ 2931.725791] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 12:31:18 (1784046678) [ 2941.002196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2941.009467] Lustre: *** cfs_fail_loc=197, val=0*** [ 2956.703715] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2961.823471] LustreError: 103484:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.107@tcp: failed processing log, type 4: rc = -110 [ 2967.042886] Lustre: *** cfs_fail_loc=197, val=0*** [ 2967.046261] Lustre: Skipped 65 previous similar messages [ 2987.615433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2987.623212] Lustre: Skipped 3 previous similar messages [ 2994.606561] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3002.439597] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3004.803498] Lustre: server umount lustre-OST0000 complete [ 3014.599330] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 12:32:42 (1784046762) [ 3036.717092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3041.978869] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3051.991928] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3056.063565] LustreError: 105136:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3056.076523] LustreError: 105136:0:(lod_sub_object.c:937:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3056.091437] LustreError: 105136:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 3056.477624] Lustre: server umount lustre-MDT0000 complete [ 3089.742617] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 3100.920187] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 3139.727303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3139.949746] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3140.008139] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3140.106523] Lustre: lustre-MDT0000: new disk, initializing [ 3140.322426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3146.503559] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3158.051305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3158.241678] Lustre: 109311:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3158.587879] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3163.705793] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3175.262728] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3183.123514] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3190.388707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3190.820572] Lustre: 111284:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3197.924538] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3200.149984] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3208.712333] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3213.898901] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3214.133544] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3219.543664] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3219.841935] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3225.572100] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3225.582305] LustreError: Skipped 3 previous similar messages [ 3225.596410] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3225.619119] Lustre: Skipped 4 previous similar messages [ 3225.634599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3225.641578] Lustre: Skipped 5 previous similar messages [ 3236.323406] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3236.647666] Lustre: server umount lustre-OST0000 complete [ 3246.892489] Lustre: server umount lustre-MDT0000 complete [ 3250.907320] LustreError: 111283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784047000 with bad export cookie 2865685391915217986 [ 3250.912128] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3250.918371] LustreError: 111283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3251.215391] Lustre: server umount lustre-MDT0001 complete [ 3260.780969] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 12:36:48 (1784047008) [ 3271.318281] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 3273.769826] Key type lgssc unregistered [ 3274.126864] LNet: 113772:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3274.140981] LNetError: 113772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3274.163370] LNet: Removed LNI 192.168.204.107@tcp [ 3275.080892] Key type .llcrypt unregistered [ 3275.083741] Key type ._llcrypt unregistered [ 3295.090734] Key type ._llcrypt registered [ 3295.097572] Key type .llcrypt registered [ 3295.235797] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 3313.892390] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 3315.572373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3315.666057] alg: No test for adler32 (adler32-zlib) [ 3316.748310] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 3317.095386] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 3318.783210] Key type lgssc registered [ 3320.095570] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3362.417723] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3362.437587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3363.656960] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3363.688614] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3363.753405] Lustre: lustre-MDT0000: new disk, initializing [ 3363.822530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3363.834245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3368.291855] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3377.752575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3378.013819] Lustre: lustre-OST0000: new disk, initializing [ 3378.017768] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3378.021244] Lustre: Skipped 1 previous similar message [ 3378.026990] Lustre: 117949:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3378.092417] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3378.941724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3378.953075] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3379.002052] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3383.716662] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3394.067529] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3394.235435] Lustre: lustre-OST0001: new disk, initializing [ 3394.238528] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3394.243295] Lustre: 118811:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3394.310628] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3400.313825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3400.319605] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3400.351245] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3400.693062] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3410.404525] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3410.420161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3410.424704] Lustre: Skipped 1 previous similar message [ 3415.164473] Lustre: server umount lustre-MDT0000 complete [ 3420.952446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3425.281529] LustreError: 117948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784047174 with bad export cookie 15222735105622675485 [ 3425.288574] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3425.292225] LustreError: 117948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3425.420098] Lustre: server umount lustre-OST0000 complete [ 3428.908566] Lustre: server umount lustre-OST0001 complete [ 3438.974727] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 12:39:46 (1784047186) [ 3463.996627] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 12:40:11 (1784047211) [ 3473.087729] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 3492.802353] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 12:40:40 (1784047240) [ 3531.527705] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 3542.766211] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 3579.572414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3579.925804] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3579.966591] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3580.094490] Lustre: lustre-MDT0000: new disk, initializing [ 3580.244127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3580.275832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3585.751335] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3597.310252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3597.455662] Lustre: 126567:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3597.518271] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3597.528434] Lustre: Skipped 1 previous similar message [ 3597.632370] Lustre: lustre-MDT0001: new disk, initializing [ 3597.862122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3597.952640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3597.982475] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3604.085495] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3613.760625] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3619.869760] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3627.516528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3627.800289] Lustre: lustre-OST0000: new disk, initializing [ 3627.816648] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3627.835330] Lustre: 128536:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3627.943343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3634.804349] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3636.807132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3636.822607] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3636.980224] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3643.859230] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3648.527237] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3648.795803] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3654.461657] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3654.718851] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3657.701093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3657.708039] Lustre: Skipped 1 previous similar message [ 3657.711451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3657.714110] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3662.877423] Lustre: server umount lustre-OST0000 complete [ 3667.942775] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3667.947359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3667.976903] Lustre: Skipped 2 previous similar messages [ 3668.000983] Lustre: Skipped 2 previous similar messages [ 3673.068180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3673.075360] Lustre: Skipped 1 previous similar message [ 3678.188668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3678.192059] Lustre: Skipped 1 previous similar message [ 3681.247392] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3681.523215] Lustre: server umount lustre-MDT0000 complete [ 3683.301493] LustreError: 128543:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3683.320798] LustreError: 128543:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3685.264633] LustreError: 126557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784047434 with bad export cookie 15222735105622676710 [ 3685.266058] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3685.275402] LustreError: 126557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3685.662615] Lustre: server umount lustre-MDT0001 complete [ 3693.810726] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 12:44:01 (1784047441) [ 3701.055959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3701.577048] LustreError: 130949:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3701.661421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3706.190556] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3706.872091] LustreError: 130950:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3710.955774] LustreError: 130949:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3715.562543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3721.258837] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3729.865435] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3736.587633] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3743.528321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3743.864174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3743.871425] Lustre: Skipped 1 previous similar message [ 3750.842754] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3758.628296] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3766.824637] Lustre: *** cfs_fail_loc=159, val=0*** [ 3766.833895] Lustre: Skipped 1 previous similar message [ 3782.094312] Lustre: lustre-MDT0000: Client 46c869a5-5a3c-49f9-a126-a944e05552fe (at 192.168.204.7@tcp) reconnecting [ 3783.670557] Lustre: *** cfs_fail_loc=159, val=0*** [ 3798.986928] Lustre: lustre-MDT0000: Client 46c869a5-5a3c-49f9-a126-a944e05552fe (at 192.168.204.7@tcp) reconnecting [ 3805.933336] Lustre: server umount lustre-OST0000 complete [ 3812.841426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3812.848972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3812.861305] Lustre: Skipped 1 previous similar message [ 3812.890809] Lustre: Skipped 1 previous similar message [ 3819.065734] Lustre: server umount lustre-MDT0000 complete [ 3822.754406] LustreError: 130932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784047571 with bad export cookie 15222735105622677592 [ 3822.759519] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3822.763881] LustreError: 130932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3832.459787] Lustre: server umount lustre-MDT0001 complete [ 3843.179540] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 3846.024202] Key type lgssc unregistered [ 3846.423913] LNet: 134734:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3846.432770] LNetError: 134734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3846.454232] LNet: Removed LNI 192.168.204.107@tcp [ 3847.257239] Key type .llcrypt unregistered [ 3847.259761] Key type ._llcrypt unregistered [ 3869.352326] Key type ._llcrypt registered [ 3869.354165] Key type .llcrypt registered [ 3869.435170] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 3870.291896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3870.332272] alg: No test for adler32 (adler32-zlib) [ 3871.451837] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 3871.696082] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 3873.344946] Key type lgssc registered [ 3874.282347] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3881.154633] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3881.179490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3882.663754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3887.254356] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3892.201000] LustreError: 136148:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3892.215251] LustreError: 136148:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3895.790310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3896.224050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3900.719419] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3908.474408] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3913.481851] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3920.380463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3920.773259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3926.928983] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3927.340478] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3936.151358] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3945.197026] Lustre: *** cfs_fail_loc=159, val=0*** [ 3961.805365] Lustre: lustre-MDT0000: Client d2b4e265-2d8e-4fae-80e8-842ffc2687cb (at 192.168.204.7@tcp) reconnecting [ 3964.696963] Lustre: *** cfs_fail_loc=159, val=0*** [ 3980.248162] Lustre: lustre-MDT0001: Client d2b4e265-2d8e-4fae-80e8-842ffc2687cb (at 192.168.204.7@tcp) reconnecting [ 3981.852115] Lustre: *** cfs_fail_loc=159, val=0*** [ 3981.866503] Lustre: Skipped 3 previous similar messages [ 3997.146956] Lustre: lustre-MDT0001: Client d2b4e265-2d8e-4fae-80e8-842ffc2687cb (at 192.168.204.7@tcp) reconnecting [ 4013.712930] Lustre: 139215:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4016.578649] Lustre: *** cfs_fail_loc=159, val=0*** [ 4016.596139] Lustre: Skipped 7 previous similar messages [ 4033.011818] Lustre: lustre-MDT0000: Client f8431554-b58b-47d1-86e2-d65c63a47aef (at 192.168.204.7@tcp) reconnecting [ 4034.900268] Lustre: *** cfs_fail_loc=159, val=0*** [ 4034.903292] Lustre: Skipped 10 previous similar messages [ 4050.370347] Lustre: lustre-MDT0000: Client f8431554-b58b-47d1-86e2-d65c63a47aef (at 192.168.204.7@tcp) reconnecting [ 4051.675957] Lustre: 139510:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4075.491456] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4075.494495] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4075.497272] LustreError: Skipped 1 previous similar message [ 4075.498051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4075.512767] Lustre: Skipped 1 previous similar message [ 4075.522031] Lustre: Skipped 1 previous similar message [ 4077.311254] Lustre: server umount lustre-OST0000 complete [ 4085.728659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4085.734259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4085.740273] Lustre: Skipped 1 previous similar message [ 4085.751404] Lustre: Skipped 1 previous similar message [ 4087.375512] Lustre: server umount lustre-MDT0000 complete [ 4090.612123] LustreError: 139338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784047839 with bad export cookie 1534649911023725450 [ 4090.616434] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4090.634538] LustreError: 139338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4090.865894] Lustre: server umount lustre-MDT0001 complete [ 4101.938193] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 4104.673811] Key type lgssc unregistered [ 4104.918446] LNet: 140590:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4104.926184] LNetError: 140590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4104.936746] LNet: Removed LNI 192.168.204.107@tcp [ 4105.761806] Key type .llcrypt unregistered [ 4105.765534] Key type ._llcrypt unregistered [ 4127.479199] Key type ._llcrypt registered [ 4127.482503] Key type .llcrypt registered [ 4127.625956] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 4128.467531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4128.484576] alg: No test for adler32 (adler32-zlib) [ 4129.584327] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 4129.827173] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 4131.536993] Key type lgssc registered [ 4132.649680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4138.946906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4138.960808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4140.378083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4144.058687] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4150.754034] LustreError: 142004:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4150.773055] LustreError: 142004:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4151.818079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4152.119680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4156.724868] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4164.525506] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4169.976221] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4177.104507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4177.540611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4184.658528] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4185.725420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4191.927854] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4197.042771] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4201.972437] Lustre: 144585:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4205.048435] Lustre: 144633:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4206.602808] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4207.625219] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4211.116611] Lustre: 144779:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4212.010900] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4212.754150] Lustre: 144878:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4216.800659] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4216.811837] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4216.825584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4217.827849] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4217.834257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4217.841881] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4222.924090] Lustre: server umount lustre-OST0000 complete [ 4228.065291] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4228.068975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4228.079927] Lustre: Skipped 1 previous similar message [ 4228.102129] Lustre: Skipped 2 previous similar messages [ 4232.818894] Lustre: server umount lustre-MDT0000 complete [ 4233.185811] LustreError: 141999:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4233.219859] LustreError: 141999:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4236.542756] LustreError: 141985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784047985 with bad export cookie 15168782920032784011 [ 4236.555348] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4236.560334] LustreError: 141985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4237.024070] Lustre: server umount lustre-MDT0001 complete [ 4248.820405] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 4251.462310] Key type lgssc unregistered [ 4251.756650] LNet: 145939:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4251.764809] LNetError: 145939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4251.783104] LNet: Removed LNI 192.168.204.107@tcp [ 4252.615186] Key type .llcrypt unregistered [ 4252.617079] Key type ._llcrypt unregistered [ 4276.832306] Key type ._llcrypt registered [ 4276.836256] Key type .llcrypt registered [ 4276.932487] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 4277.715426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4277.795077] alg: No test for adler32 (adler32-zlib) [ 4278.840633] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 4279.090335] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 4280.751700] Key type lgssc registered [ 4281.725238] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4287.921791] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4287.959468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4289.385704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4292.849737] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4299.631075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4299.751754] LustreError: 147352:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4299.776636] LustreError: 147352:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4299.961301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4303.505837] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4310.237100] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4315.881931] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4322.678472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4323.186470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4325.241211] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4329.138177] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4334.466235] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4336.821248] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4342.197753] Lustre: *** cfs_fail_loc=159, val=0*** [ 4358.626185] Lustre: lustre-MDT0000: Client a9626c9f-f147-4cda-b4b2-7d63b20956e7 (at 192.168.204.7@tcp) reconnecting [ 4375.521753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4375.523072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4375.553693] Lustre: Skipped 1 previous similar message [ 4380.646454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4380.653619] Lustre: Skipped 2 previous similar messages [ 4381.568842] Lustre: server umount lustre-OST0000 complete [ 4385.760513] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4385.762495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4385.770703] Lustre: Skipped 1 previous similar message [ 4385.785613] Lustre: Skipped 1 previous similar message [ 4390.883370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4390.895368] Lustre: Skipped 1 previous similar message [ 4391.592310] Lustre: server umount lustre-MDT0000 complete [ 4394.623571] LustreError: 149967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784048143 with bad export cookie 11717487166625852077 [ 4394.627774] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4394.631044] LustreError: 149967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4394.859965] Lustre: server umount lustre-MDT0001 complete [ 4403.541717] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 4405.555303] Key type lgssc unregistered [ 4405.751368] LNet: 151021:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4405.763979] LNetError: 151021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4406.826912] LNet: Removed LNI 192.168.204.107@tcp [ 4407.486170] Key type .llcrypt unregistered [ 4407.487745] Key type ._llcrypt unregistered [ 4431.016555] Key type ._llcrypt registered [ 4431.018841] Key type .llcrypt registered [ 4431.132280] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 4432.413732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4432.441387] alg: No test for adler32 (adler32-zlib) [ 4433.437712] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 4433.633856] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 4435.311173] Key type lgssc registered [ 4436.206937] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4442.940143] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4442.983560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4444.489879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4448.968222] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4453.862449] LustreError: 152436:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4453.894374] LustreError: 152436:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4457.083242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4457.516988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4461.946757] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4469.165982] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4474.380756] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4480.399557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4480.613036] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4482.671119] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4485.915548] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4489.839920] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4493.476678] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4513.417045] Lustre: lustre-MDT0001: evicting 20d7b732-3a98-473b-bbe9-02cec1b374be (at 192.168.204.7@tcp) by administrative request [ 4513.424523] Lustre: Skipped 1 previous similar message [ 4541.411365] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4541.424828] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4541.434209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4541.444591] Lustre: Skipped 1 previous similar message [ 4541.470075] Lustre: Skipped 1 previous similar message [ 4546.535691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4547.466045] Lustre: server umount lustre-OST0000 complete [ 4554.726574] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4554.736103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4554.746285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4554.754985] Lustre: Skipped 1 previous similar message [ 4556.777029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4556.777168] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4557.956861] Lustre: server umount lustre-MDT0000 complete [ 4561.891135] LustreError: 153167:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4561.914086] LustreError: 153167:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4561.978424] LustreError: 152416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784048311 with bad export cookie 17146911330776518759 [ 4561.983698] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4561.987198] LustreError: 152416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4562.348468] Lustre: server umount lustre-MDT0001 complete [ 4574.750067] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 4577.897945] Key type lgssc unregistered [ 4578.250628] LNet: 156502:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4578.268643] LNetError: 156502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4578.310460] LNet: Removed LNI 192.168.204.107@tcp [ 4579.133425] Key type .llcrypt unregistered [ 4579.136316] Key type ._llcrypt unregistered [ 4656.904035] Key type ._llcrypt registered [ 4656.904978] Key type .llcrypt registered [ 4656.998601] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 4657.696865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4657.709552] alg: No test for adler32 (adler32-zlib) [ 4658.747285] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 4658.963127] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 4660.647174] Key type lgssc registered [ 4661.858982] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4669.595098] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4669.631786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4671.253865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4676.524779] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4681.697336] LustreError: 158992:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4681.727438] LustreError: 158992:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4685.248879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4685.816237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4690.470849] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4698.187703] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4704.414691] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4711.610544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4712.021633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4716.156932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 4718.281299] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4726.407480] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4730.560893] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 4741.093731] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4741.094168] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4741.097925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4741.123813] Lustre: Skipped 1 previous similar message [ 4746.214120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4746.219305] Lustre: Skipped 2 previous similar messages [ 4746.935468] Lustre: server umount lustre-OST0000 complete [ 4752.354439] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4752.363179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4752.372557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4756.453023] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4756.453387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4756.475440] Lustre: Skipped 1 previous similar message [ 4757.870639] Lustre: server umount lustre-MDT0000 complete [ 4761.570724] LustreError: 158992:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4761.595418] LustreError: 158992:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4762.093251] LustreError: 158973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784048511 with bad export cookie 12646841909919635270 [ 4762.101237] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4762.105281] LustreError: 158973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4762.428970] Lustre: server umount lustre-MDT0001 complete [ 4774.959910] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 4778.093833] Key type lgssc unregistered [ 4778.403763] LNet: 162566:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4778.411377] LNetError: 162566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4778.434937] LNet: Removed LNI 192.168.204.107@tcp [ 4779.367888] Key type .llcrypt unregistered [ 4779.375390] Key type ._llcrypt unregistered [ 4808.744196] Key type ._llcrypt registered [ 4808.752194] Key type .llcrypt registered [ 4808.983743] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 4811.180392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4811.226146] alg: No test for adler32 (adler32-zlib) [ 4812.395091] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 4812.797991] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 4814.642298] Key type lgssc registered [ 4816.166168] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4833.717858] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 13:03:01 (1784048581) [ 4851.405996] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 4861.876350] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 4897.432477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4897.445791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4898.592215] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4898.617340] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4898.669839] Lustre: lustre-MDT0000: new disk, initializing [ 4898.724092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4898.740147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4902.839310] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4914.441446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4914.688066] Lustre: 166832:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4914.768865] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4914.772324] Lustre: Skipped 1 previous similar message [ 4914.946230] Lustre: lustre-MDT0001: new disk, initializing [ 4915.155933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4915.210487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4915.230425] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4920.108706] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4930.002455] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4935.552715] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4942.106058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4942.379439] Lustre: lustre-OST0000: new disk, initializing [ 4942.385826] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4942.394538] Lustre: 168800:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4942.480096] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4948.906783] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4950.582498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4950.590146] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4950.740275] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4957.906360] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4971.492309] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4971.496255] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4971.516219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4971.529500] Lustre: Skipped 1 previous similar message [ 4976.612023] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4976.626537] Lustre: Skipped 1 previous similar message [ 4981.739046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4981.747411] Lustre: Skipped 1 previous similar message [ 4982.244692] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4982.528288] Lustre: server umount lustre-OST0000 complete [ 4986.849275] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4986.850688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4986.850934] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4986.867670] Lustre: Skipped 2 previous similar messages [ 4986.903445] Lustre: Skipped 1 previous similar message [ 4991.968672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4991.982653] Lustre: Skipped 1 previous similar message [ 4992.763238] Lustre: server umount lustre-MDT0000 complete [ 4996.781943] LustreError: 166823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784048745 with bad export cookie 10111979121617934785 [ 4996.782204] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4996.798040] LustreError: 166823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4997.174530] Lustre: server umount lustre-MDT0001 complete [ 5009.338349] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 5011.985178] Key type lgssc unregistered [ 5012.235085] LNet: 170706:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5012.245898] LNetError: 170706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5012.258882] LNet: Removed LNI 192.168.204.107@tcp [ 5013.022954] Key type .llcrypt unregistered [ 5013.024807] Key type ._llcrypt unregistered [ 5039.314960] Key type ._llcrypt registered [ 5039.318133] Key type .llcrypt registered [ 5039.462667] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 5040.318796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5040.423043] alg: No test for adler32 (adler32-zlib) [ 5041.628901] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 5041.884115] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 5043.583170] Key type lgssc registered [ 5044.776639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5051.034788] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5051.059608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5052.592534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5058.212473] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5062.120750] LustreError: 172122:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5062.140063] LustreError: 172122:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5067.236765] LustreError: 172121:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5071.848517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5072.444643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5078.207414] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5086.178941] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5092.525621] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5099.812599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5100.139441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5106.104656] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5113.530106] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5123.694608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5129.689899] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5129.702787] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5129.725564] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5129.734265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5129.749783] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5130.655924] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5130.699813] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5130.726702] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5130.740226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5130.768777] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5131.564900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5132.596546] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5132.611912] Lustre: Skipped 1 previous similar message [ 5132.642144] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5132.654601] Lustre: Skipped 1 previous similar message [ 5132.663511] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5132.678273] Lustre: Skipped 1 previous similar message [ 5132.680901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5132.695962] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5132.705490] Lustre: Skipped 1 previous similar message [ 5133.628467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5134.566449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5135.468646] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5135.477600] Lustre: Skipped 2 previous similar messages [ 5135.494288] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5135.505887] Lustre: Skipped 2 previous similar messages [ 5135.510894] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5135.516190] Lustre: Skipped 2 previous similar messages [ 5135.520840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5135.532592] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5135.541707] Lustre: Skipped 2 previous similar messages [ 5136.527815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5137.516846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5138.518055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5139.554276] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5139.562063] Lustre: Skipped 3 previous similar messages [ 5139.587326] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5139.599251] Lustre: Skipped 3 previous similar messages [ 5139.603775] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5139.618027] Lustre: Skipped 3 previous similar messages [ 5139.622553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5139.631260] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5139.638489] Lustre: Skipped 3 previous similar messages [ 5140.377576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5141.145299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5142.163791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5143.056147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5144.092774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5145.115862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5146.088701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5147.110485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5148.114256] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5148.126111] Lustre: Skipped 8 previous similar messages [ 5148.139967] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5148.157150] Lustre: Skipped 8 previous similar messages [ 5148.162726] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5148.168221] Lustre: Skipped 8 previous similar messages [ 5148.172183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5148.181716] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5148.192953] Lustre: Skipped 8 previous similar messages [ 5149.028618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5150.055928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5150.969864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5151.823554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5152.733802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5153.618865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5154.482718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5155.507497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5156.430696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5157.449297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5158.385855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5159.302716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5160.322122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5161.261583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5162.169359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5163.128738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5163.983798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5164.765038] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5164.776513] Lustre: Skipped 17 previous similar messages [ 5164.799086] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5164.811852] Lustre: Skipped 17 previous similar messages [ 5164.815409] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5164.822544] Lustre: Skipped 17 previous similar messages [ 5164.827385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5164.840990] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5164.852045] Lustre: Skipped 17 previous similar messages [ 5165.690871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5166.601648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5167.478220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5168.480449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5169.298690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5170.170990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5171.020666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5171.862193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5172.654677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5173.611189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5174.560038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5175.473693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5176.476595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5177.287533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5178.136455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5178.884617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5179.818880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5180.699664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5181.674203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5182.511535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5183.503361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5184.584061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5185.843288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5186.860810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5188.004142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5189.000591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5189.792225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5190.599683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5191.618808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5192.600335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5193.669688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5194.704255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5195.938439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5196.860446] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5196.889206] Lustre: Skipped 33 previous similar messages [ 5196.926070] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5196.944416] Lustre: Skipped 33 previous similar messages [ 5196.956778] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5196.967658] Lustre: Skipped 33 previous similar messages [ 5196.973617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5196.990711] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5197.002213] Lustre: Skipped 33 previous similar messages [ 5198.085545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5199.270143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5200.065778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5201.149838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5202.304770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5203.255769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5204.350738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5205.157786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5206.219214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5207.080488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5207.989652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5208.872527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5209.975676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5210.871102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5211.787759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5212.610391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5213.662634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5214.546383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5215.355223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5216.194136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5217.077347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5217.777033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5218.572058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5219.468575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5220.246521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5221.157841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5221.980582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5222.912567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5223.943320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5224.891598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5225.868990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5226.708887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5227.507098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5228.345219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5229.165949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5229.862515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5230.688385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5231.656317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5232.687954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5233.680940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5234.478955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5235.310331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5236.179304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5237.029838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5237.949928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5238.863439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5239.776088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5240.743897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5241.518032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5242.362894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5243.156954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5243.943869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5244.827743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5245.630904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5246.369706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5247.196875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5247.994120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5248.815477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5249.601304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5250.351608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5250.991816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5251.753580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5252.526646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5253.313415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5254.106086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5255.009518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5255.831574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5256.726833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5257.661351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5258.711865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5259.605418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5260.532902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5261.285301] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5261.289450] Lustre: Skipped 72 previous similar messages [ 5261.299755] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5261.306637] Lustre: Skipped 72 previous similar messages [ 5261.309908] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5261.313621] Lustre: Skipped 72 previous similar messages [ 5261.316605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5261.324425] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5261.331289] Lustre: Skipped 72 previous similar messages [ 5262.024701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5262.861704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5263.719813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5264.565586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5265.466765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5266.318060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5267.052751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5267.850314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5268.584794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5269.347644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5270.189564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5270.989856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5271.803072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5272.677322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5273.513948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5274.370321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5275.128518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5275.906402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5276.842871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5277.641599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5278.435261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5279.183238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5279.961432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5280.865978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5281.880904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5282.921794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5283.816721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5284.719066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5285.666813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5286.556797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5287.490110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5288.299816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5289.325495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5290.254778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5291.259241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5292.156424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5293.170733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5294.170238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5295.178567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5295.966340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5296.797373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5297.545704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5298.482913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5299.356835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5300.224445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5301.014561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5301.810627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5302.575163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5303.494535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5304.348304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5305.284816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5306.150764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5307.033875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5307.896042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5308.852647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5309.737418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5310.745316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5311.654927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5312.700968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5313.576383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5314.485267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5315.288936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5316.089072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5316.820091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5317.671534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5318.533700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5319.443516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5320.113686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5321.230629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5322.021409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5322.847859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5323.722729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5324.566556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5325.235520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5326.034829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5326.916867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5327.786516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5328.620993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5329.351276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5330.083788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5330.964265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5331.746108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5332.773843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5333.696034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5334.651413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5335.519300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5336.524517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5337.432286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5338.196080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5338.918209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5339.832922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5340.690588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5341.472213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5342.354173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5343.549821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5344.576581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5345.659426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5346.592375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5347.637423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5348.538914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5349.373652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5350.052100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5350.863604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5351.706094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5352.572789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5353.580305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5354.481156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5355.335482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5356.303149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5357.179846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5358.071537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5359.005524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5359.910269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5360.793271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5361.622743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5362.478511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5363.406030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5364.311830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5365.370885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5366.395213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5367.299676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5368.308777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5369.105705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5370.047393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5370.926156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5371.801595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5372.753664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5373.544409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5374.582169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5375.374908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5376.252611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5377.118719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5378.052333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5378.855431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5379.775246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5380.764738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5381.711043] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5382.629386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5383.575119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5384.437304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5385.220318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5386.154901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5387.127365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5388.004060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5388.925580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5389.797122] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5389.803281] Lustre: Skipped 145 previous similar messages [ 5389.820771] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5389.827953] Lustre: Skipped 145 previous similar messages [ 5389.834725] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5389.842184] Lustre: Skipped 145 previous similar messages [ 5389.850267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5389.864203] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5389.878851] Lustre: Skipped 145 previous similar messages [ 5390.776237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5391.708419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5392.657163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5393.661628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5394.460780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5395.283226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5396.255504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.112827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.966111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5398.870082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5399.795664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5400.601085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5401.481138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5402.225640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5403.044046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5403.825454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5404.652835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5405.528684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5406.297692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5407.235704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.091418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.753326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5409.619396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5410.339795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5411.194549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5412.035275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5413.062047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5413.967630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5414.961677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5415.808546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5416.617646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5417.416371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.221827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.939113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5419.653824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5420.342139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5421.265276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5422.037345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5422.811048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5423.724416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5424.742664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5425.791054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5426.851499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5427.908982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5428.779777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5429.772576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5430.554840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5431.490047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5432.466780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5433.398113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5434.302426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5435.011036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5436.068503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5436.955344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5437.890626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5438.881387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5439.925809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5441.133222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5442.220509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5443.511069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5444.445957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5445.489944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5446.808458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5447.896990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5449.045025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5450.293857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5451.256360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5452.377606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5453.587271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5454.696603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5455.605229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5456.704745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5457.624281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5458.604384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5459.629704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5460.597800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.595086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5462.568338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5463.489247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5464.416990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5465.170663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5466.172705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.198765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5468.109966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.051705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5470.002582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5471.018640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5472.065547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5472.991031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5474.023789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5474.985593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.902510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5476.791906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5477.683571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5478.557679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5479.368957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5480.223752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.010959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.872265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.801191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5483.547514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.444956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.290882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5486.219129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5487.200666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.132568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.976860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.858640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5490.893335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.912481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.141724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5494.195599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.198774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5496.358471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.710886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5498.859960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.895894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.972796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5502.310358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5503.382582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.432146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.447341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.524683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.471659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.479984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5509.311633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.345960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5511.288232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.334753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.434976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5514.407604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.329408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5516.176562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.134731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.968786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.973278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.012607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.111623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5522.028251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.183254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.305837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5525.519679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5526.542633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.741123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.711080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.680465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5530.693759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.830485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.673524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5533.765255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.704779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5535.728892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.606476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5537.726515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.733483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5539.714774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.621521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5541.549331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.761962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.915257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.858089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.007784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.001138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.959612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.822399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.853819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.908840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.959651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.959928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.919898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.936110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.886787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5556.831792] LustreError: 172141:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5556.847633] LustreError: 172141:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5556.856639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.730521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.877865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.809575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.736439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.781787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.662334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.780128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.905819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.020450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.951686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.949451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.916750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.068570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.133165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.105459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.302507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.580237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.751778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.945091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.488764] LustreError: 172141:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5578.501252] LustreError: 172141:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5578.533156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.808900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.808117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.040375] LustreError: 172141:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5582.056514] LustreError: 172141:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5582.071492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.275919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.298955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.410898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5586.626511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.853000] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.918642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.938591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.152460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.150695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.216967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.275414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.481376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.394257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.260669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.435508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.772358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.038490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.345101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.576988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.743637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.001429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.230389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.568796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.827204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.861218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.900278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.132936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.134379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.337502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.578083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.096943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.271147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.521722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.797049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.931948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.186522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.314257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.410363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.412681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.552576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.681455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.826683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.798544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.111395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.031408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.929902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.557549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.198383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.841617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.528109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.249750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.851921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.463387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.110699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.708816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.289744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.844579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.453138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.967973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.481428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.963260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.473801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.043388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.677296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.171590] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5646.174970] Lustre: Skipped 260 previous similar messages [ 5646.183597] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5646.187673] Lustre: Skipped 260 previous similar messages [ 5646.189634] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5646.191651] Lustre: Skipped 260 previous similar messages [ 5646.193460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.198165] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5646.200301] Lustre: Skipped 260 previous similar messages [ 5646.758465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.325743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.173769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.519161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.844212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.857270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.290679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.303642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.858396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.982331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.992658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.085094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.500574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.829436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.269255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.654452] LustreError: 172141:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5664.663529] LustreError: 172141:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5664.685708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.675737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.224363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.003270] LustreError: 172141:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5669.017651] LustreError: 172141:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5669.036400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.806994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.071368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.287949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.603609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.719893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.957996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.672371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.989557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.721470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.396466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.317061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.404962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.525537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.675187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.745147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.705376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.569904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.460708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.289766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.122936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.077741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.119836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.117885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.146123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.206870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.136475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.371500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.609416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.522867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.585056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.501866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.621284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.740304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.704690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.645370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.790419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.618492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5712.701825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.589918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.718755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.731886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.913769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.076308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.321092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.445574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.611650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.634996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.745769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.860788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.790595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.777755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.756044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.791256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.968512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.845768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.885336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.729681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.667416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.608639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.894817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.835306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.947375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.932538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.753947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5740.624941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.530569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.490820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.644226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.491686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.522996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.368723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.255399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.062750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.923314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.737846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5750.561428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.370274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.120723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.859704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.597693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.383639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.283740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.163515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.066222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.845358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.786875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.618852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.537472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.486504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.637311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.617509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.841943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.755980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.823985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.934881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.226896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.102851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.958529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.902662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.953935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.974273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.958145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.888059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.961460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.988670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.980427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.005162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.126078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.123759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.338108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.538098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.615866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.732824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.785988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.851772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.733198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.706663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.591302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.529140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.430976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.281619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.368775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.273961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.146587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.790366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.639792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.411553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.173468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.975447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.812825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.515583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.357224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.193194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.042638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.846496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.732901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.690478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.585671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.391606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.293726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.434027] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.298651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.187250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.979723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.013941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.913886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.886332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.807480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.707394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.557855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.396534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.170968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.100407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.015520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.779840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.555921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.256220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.213317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.242903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.484829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.560334] LustreError: 172141:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5829.569057] LustreError: 172141:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5829.574871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.720197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.767698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.778848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.045466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5835.272328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.221958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.319682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.337619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.373223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.395332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.570405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.695643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.722151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5844.482221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.278274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.452071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.559165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.400906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.853972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.126871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.313093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.419056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.278291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.604158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.608102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5857.828813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.738416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.569040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.402717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.521262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.377874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.254060] LustreError: 172141:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5863.260094] LustreError: 172141:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5863.276849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.044068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.843759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.606153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.629532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.682803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.849327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.877351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.861578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.068841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.377885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.459419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5875.752189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.825690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.785730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.862398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.101472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5881.230076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5882.166388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.038467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.034153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.016681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.864609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.765460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.747868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.706786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.575190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.392355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.252695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.114657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.961238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.944924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.929215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.833230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.731465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.412680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.313536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.216283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.119987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.972542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5901.824783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.666603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.500272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.441869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.308023] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.142227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.032675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.779882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.668808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.553342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.312881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.169364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.094532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.170400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5914.145787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.108334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.931504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.100034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.020186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5919.179894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.171019] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5921.234487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.211955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5923.107758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5923.896272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.822913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.692478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.558472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5927.436843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5928.616087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5929.652842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5930.549236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.555435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.447813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5933.403179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5934.286327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5935.387897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5936.668205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5937.673589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5938.603713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.556808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.489353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5941.194847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.021982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.864168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5943.748591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5944.658222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5945.736464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.696955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5947.718559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.747721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.852414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5950.814466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5951.946516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5952.896156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.925671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5954.970247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5956.041729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5957.133291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5958.277836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5959.545964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5960.699148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5961.862994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5962.925846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5963.765390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5964.625531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5965.568735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5966.450375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.406314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5968.345388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5969.275689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5970.250769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5971.121228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5972.271047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5973.360771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5974.328934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.468629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.526874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5977.613751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.601861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.676809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5980.649120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5981.871380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5982.996732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.102312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5985.119380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.225914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5987.434533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.463515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5989.583420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5990.575865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.563646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5992.469971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5993.430240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5994.354367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5995.334937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5996.594161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5997.697479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.754887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.045983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6001.159773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6002.303464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6003.626395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6004.742760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6005.924438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6007.188271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6008.404093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6009.496490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6010.683710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6011.675952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6012.747652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.144557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6015.749208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6017.138119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6018.866478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6019.954308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6021.028893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6022.373056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6023.766466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6025.138875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6026.327905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6027.466925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6028.425851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6029.460203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6030.507326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6031.566191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6032.515150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6033.616320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.418617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6035.485454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6036.365369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6037.488817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.507703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6039.528197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6040.530190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6041.563867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6042.654682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6043.860020] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6045.205219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6046.179596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6047.178686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6048.314836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6049.499104] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6050.456882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6051.631959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6052.660237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6053.797957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6054.905740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6055.908484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6057.019840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6057.928181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6058.999589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6059.878454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6060.771843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6061.610203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6062.579213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6063.458991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.534929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6065.402581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6066.243915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6067.059286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6067.871550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6068.792334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6069.677370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6070.447551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6071.437310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6072.271586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6073.355176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6074.446572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.605680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6076.802773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6077.838199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6078.879336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6080.020278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6081.040506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6082.137170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6083.102876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6084.235231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6085.331433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6086.489638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6087.516235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6088.573415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6089.363172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6090.496573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6091.512531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6092.703264] LustreError: 172141:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6092.722653] LustreError: 172141:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6092.773518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6094.496947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6096.068594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6097.311763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6098.954622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6100.122200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6101.261458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6103.241753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6104.691845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6106.554290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6107.704891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6108.529445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6614.290706] LustreError: 173942:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6614.296542] LustreError: 172141:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 6614.305970] LustreError: 172141:0:(osp_precreate.c:655:osp_precreate_send()) Skipped 1 previous similar message [ 7401.440220] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7401.449539] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7401.461837] Lustre: Skipped 448 previous similar messages [ 7401.471580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7402.976645] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7402.992739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7404.521602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7404.531921] Lustre: Skipped 1 previous similar message [ 7407.767201] Lustre: server umount lustre-OST0000 complete [ 7412.193072] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7412.206084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7418.047442] Lustre: server umount lustre-MDT0000 complete [ 7419.873474] LustreError: 184069:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7419.904843] LustreError: 184069:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7422.134501] LustreError: 172102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784051171 with bad export cookie 10515722776022957662 [ 7422.140303] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7422.150368] LustreError: 172102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7422.622778] Lustre: server umount lustre-MDT0001 complete [ 7434.292204] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 7437.046887] Key type lgssc unregistered [ 7437.345914] LNet: 227439:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7437.371622] LNetError: 227439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7437.386552] LNet: Removed LNI 192.168.204.107@tcp [ 7438.357094] Key type .llcrypt unregistered [ 7438.360222] Key type ._llcrypt unregistered [ 7465.107454] Key type ._llcrypt registered [ 7465.109618] Key type .llcrypt registered [ 7465.245411] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 7466.362236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7466.421471] alg: No test for adler32 (adler32-zlib) [ 7467.636230] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 7468.018440] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 7469.759196] Key type lgssc registered [ 7471.294942] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7479.650168] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7479.668616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7481.189539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7484.888814] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7491.557391] LustreError: 228857:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7491.585505] LustreError: 228857:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7495.086726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7495.481623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7501.193773] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7510.324591] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7517.789736] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7526.108629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7526.403982] LustreError: 230679:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7526.910570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7532.057449] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 7534.493408] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7543.780668] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7549.096285] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7551.695346] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7551.706164] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 7551.715035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7551.725085] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7551.725883] LustreError: 228877:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 7555.554415] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7555.572097] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7555.589555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7557.610980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7557.617972] Lustre: Skipped 1 previous similar message [ 7557.619699] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7560.377681] Lustre: server umount lustre-OST0000 complete [ 7571.735988] LustreError: 228877:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 7595.750040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7595.936701] LustreError: 232064:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7596.201188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7597.240913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7597.255497] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7604.689550] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7607.473305] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7607.488914] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7613.040571] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7615.001157] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7620.945109] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff98ca46a41000.ost_server_uuid 50 [ 7622.205937] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff98ca46a41000.ost_server_uuid in FULL state after 0 sec [ 7627.750837] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7627.757734] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7627.772031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7627.780793] Lustre: Skipped 1 previous similar message [ 7632.707705] Lustre: server umount lustre-OST0000 complete [ 7638.499779] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7638.505133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7638.511934] Lustre: Skipped 2 previous similar messages [ 7638.525974] Lustre: Skipped 3 previous similar messages [ 7643.414097] Lustre: server umount lustre-MDT0000 complete [ 7643.616405] LustreError: 228853:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7643.635107] LustreError: 228853:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7647.215790] LustreError: 229593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784051396 with bad export cookie 3435395596037125971 [ 7647.226720] LustreError: 229593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7647.230228] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7647.670217] Lustre: server umount lustre-MDT0001 complete [ 7657.714332] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 7660.226646] Key type lgssc unregistered [ 7660.549873] LNet: 234065:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7660.556574] LNetError: 234065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7660.570454] LNet: Removed LNI 192.168.204.107@tcp [ 7661.317145] Key type .llcrypt unregistered [ 7661.319555] Key type ._llcrypt unregistered [ 7692.345077] Key type ._llcrypt registered [ 7692.346948] Key type .llcrypt registered [ 7692.482615] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 7693.405721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7693.482257] alg: No test for adler32 (adler32-zlib) [ 7694.500365] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 7694.807893] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 7696.455385] Key type lgssc registered [ 7697.665188] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7707.320799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7707.336372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7708.759488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7713.304934] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7717.874925] LustreError: 235795:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7717.886138] LustreError: 235795:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7721.413437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7721.683375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7725.747905] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7728.386067] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7736.472789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7737.065093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7744.480704] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7753.463694] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 7754.071610] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7754.388580] Lustre: lustre-OST0001: new disk, initializing [ 7754.394345] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7754.403677] Lustre: 238102:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7754.520256] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7762.572334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7762.581400] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7762.588973] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7762.689541] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7773.620583] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7781.908199] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7797.758901] Lustre: server umount lustre-MDT0000 complete [ 7798.752531] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7798.766762] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7798.773219] LustreError: 238884:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7798.826964] Lustre: Skipped 3 previous similar messages [ 7802.099323] LustreError: 235777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784051551 with bad export cookie 5710115718967797794 [ 7802.105252] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7802.108954] LustreError: 235777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7802.556724] Lustre: server umount lustre-MDT0001 complete [ 7817.490857] Lustre: server umount lustre-OST0000 complete [ 7820.319794] Lustre: 235060:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784051553/real 1784051553] req@ffff93e444e59c00 x1870713525987072/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784051569 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7820.353025] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7821.898675] Lustre: server umount lustre-OST0001 complete [ 7845.975720] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 7857.413604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7857.917575] LustreError: 241212:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7857.942467] LustreError: 241212:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7858.030153] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7862.748795] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7863.276226] LustreError: 241213:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7863.309456] LustreError: 241213:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7868.391621] LustreError: 241212:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7871.065830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7871.603601] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7876.215324] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7879.189106] Lustre: 242355:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7885.916665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7892.792404] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7896.500568] LustreError: 242710:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7901.778918] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7902.086941] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7902.095752] Lustre: Skipped 1 previous similar message [ 7903.153970] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49730 to 0x280000401:49760) [ 7903.168199] Lustre: mylustre-OST0001: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 7909.343228] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7917.307751] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7921.202727] Lustre: 244225:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7932.898376] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7932.915632] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7932.935648] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7933.922610] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7937.912983] Lustre: server umount mylustre-MDT0000 complete [ 7939.048339] LustreError: 244531:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7939.092694] LustreError: 244531:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7942.044509] LustreError: 241187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784051691 with bad export cookie 5710115718967800601 [ 7942.051838] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7942.071109] LustreError: 241187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7942.551498] Lustre: server umount mylustre-MDT0001 complete [ 7957.267420] Lustre: server umount mylustre-OST0000 complete [ 7960.159827] Lustre: 235059:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784051693/real 1784051693] req@ffff93e54a7d1880 x1870713526047360/t0(0) o400->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784051709 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7960.190657] Lustre: mylustre-MDT0001-lwp-OST0001: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7960.206310] Lustre: Skipped 3 previous similar messages [ 7961.831660] Lustre: server umount mylustre-OST0001 complete [ 7984.920540] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 7996.362997] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7996.836826] LustreError: 246531:0:(ldlm_lib.c:1192:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7996.956740] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8002.049418] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8011.620074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8017.503695] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8021.155269] Lustre: 247674:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8028.452453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8035.512390] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8042.187618] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49730 to 0x280000401:49792) [ 8045.071037] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8050.681275] Lustre: tfs-OST0001: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 8052.513990] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8060.633982] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8064.292717] Lustre: 249548:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8073.696578] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8073.713490] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8073.738543] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 8073.743739] Lustre: Skipped 3 previous similar messages [ 8074.548339] Lustre: server umount tfs-MDT0000 complete [ 8076.259388] LustreError: 248051:0:(ldlm_lib.c:1192:target_handle_connect()) tfs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8076.290594] LustreError: 248051:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 8078.754236] LustreError: 246508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784051827 with bad export cookie 5710115718967802484 [ 8078.756252] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8078.766606] LustreError: 246508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8079.145766] Lustre: server umount tfs-MDT0001 complete [ 8093.755430] Lustre: server umount tfs-OST0000 complete [ 8097.570667] Lustre: 235057:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784051830/real 1784051830] req@ffff93e5449db480 x1870713526098048/t0(0) o400->tfs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784051846 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8097.626530] Lustre: tfs-MDT0001-lwp-OST0001: Connection to tfs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8097.647524] Lustre: Skipped 3 previous similar messages [ 8098.296716] Lustre: server umount tfs-OST0001 complete [ 8122.790694] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 8134.807832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8135.364662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8135.376704] Lustre: Skipped 3 previous similar messages [ 8139.929178] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8149.079462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8154.327764] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8157.439148] Lustre: 252937:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8164.882762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8172.381749] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8181.761169] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49730 to 0x280000401:49824) [ 8182.056330] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8187.411119] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 8189.455532] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8197.699651] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8201.865230] Lustre: 254809:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8210.634592] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 13:59:17 (1784051957) [ 8228.322802] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8228.327592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8228.353131] Lustre: Skipped 2 previous similar messages [ 8228.360520] Lustre: Skipped 1 previous similar message [ 8233.441717] LustreError: 252529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8233.468516] LustreError: 252529:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 8233.500398] Lustre: server umount lustre-MDT0000 complete [ 8237.565588] LustreError: 251773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784051986 with bad export cookie 5710115718967804157 [ 8237.571086] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8237.580281] LustreError: 251773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8238.577980] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8238.588527] Lustre: Skipped 1 previous similar message [ 8244.012481] Lustre: server umount lustre-MDT0001 complete [ 8253.582768] Lustre: server umount lustre-OST0000 complete [ 8263.964824] Lustre: server umount lustre-OST0001 complete [ 8272.153422] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 8283.961789] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 8325.839568] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 8325.839568] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 8325.839568] [ 8325.868465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 8326.137876] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8326.197385] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8326.260210] Lustre: lustre-MDT0000: new disk, initializing [ 8326.382613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8326.406595] Lustre: Skipped 3 previous similar messages [ 8326.442793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8332.409256] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8344.531644] LDISKFS-fs (dm-1): Mount option "noacl" will be removed by 3.5 [ 8344.531644] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 8344.531644] [ 8344.566510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 8344.718852] Lustre: 258007:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8344.770283] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8344.773960] Lustre: Skipped 1 previous similar message [ 8344.883992] Lustre: lustre-MDT0001: new disk, initializing [ 8345.019609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8345.036745] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8349.735328] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8360.834567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8361.082994] Lustre: lustre-OST0000: new disk, initializing [ 8361.087317] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8361.096856] Lustre: 259592:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8363.020182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8363.031506] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8363.190279] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8367.619077] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8378.011767] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8378.251559] Lustre: lustre-OST0001: new disk, initializing [ 8378.254924] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8378.263239] Lustre: 260461:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8379.537748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8379.550440] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8379.672540] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8384.951338] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8397.159901] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 14:02:24 (1784052144) [ 8412.351325] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 14:02:40 (1784052160) [ 8429.538489] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8429.550139] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8429.564705] Lustre: Skipped 3 previous similar messages [ 8429.572776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8429.579083] Lustre: Skipped 2 previous similar messages [ 8433.341818] Lustre: server umount lustre-OST0000 complete [ 8437.216509] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8437.219606] LustreError: Skipped 1 previous similar message [ 8443.393383] Lustre: server umount lustre-MDT0000 complete [ 8448.160320] LustreError: 259591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784052197 with bad export cookie 5710115718967805704 [ 8448.167651] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8448.176552] LustreError: 259591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8451.560303] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8451.563350] Lustre: Skipped 7 previous similar messages [ 8455.071192] Lustre: server umount lustre-MDT0001 complete [ 8470.203364] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 8480.863539] Lustre: server umount lustre-OST0001 complete [ 8486.836273] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 8502.066891] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 8542.059366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8542.277943] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8542.304266] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8542.379194] Lustre: lustre-MDT0000: new disk, initializing [ 8542.531512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8547.559496] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8558.895896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8559.008371] Lustre: 265018:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8559.075096] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8559.079996] Lustre: Skipped 1 previous similar message [ 8559.163243] Lustre: lustre-MDT0001: new disk, initializing [ 8559.296521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8559.313673] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8564.002419] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8573.732487] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8579.560828] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8586.827314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8587.089980] Lustre: 266991:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8587.170402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8587.176573] Lustre: Skipped 5 previous similar messages [ 8588.323693] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8588.496147] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8593.912346] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8605.189774] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8614.368180] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8614.373527] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8614.389401] LustreError: Skipped 1 previous similar message [ 8614.392541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8614.392549] Lustre: Skipped 1 previous similar message [ 8614.427132] Lustre: Skipped 6 previous similar messages [ 8618.245883] Lustre: server umount lustre-OST0000 complete [ 8628.642921] Lustre: server umount lustre-MDT0000 complete [ 8629.738958] LustreError: 265896:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8629.759374] LustreError: 265896:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 8632.388794] LustreError: 265011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784052381 with bad export cookie 5710115718967807181 [ 8632.389916] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8632.396351] LustreError: 265011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8632.751439] Lustre: server umount lustre-MDT0001 complete [ 8643.023757] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 8645.777683] Key type lgssc unregistered [ 8646.080204] LNet: 268750:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8646.090320] LNetError: 268750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8646.110745] LNet: Removed LNI 192.168.204.107@tcp [ 8647.019279] Key type .llcrypt unregistered [ 8647.025084] Key type ._llcrypt unregistered [ 8673.108185] Key type ._llcrypt registered [ 8673.111864] Key type .llcrypt registered [ 8673.259310] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 8674.479647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8674.532292] alg: No test for adler32 (adler32-zlib) [ 8675.695995] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 8676.030780] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 8677.815508] Key type lgssc registered [ 8679.076317] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8688.825297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8688.841131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8690.473614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8695.086593] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8699.880588] LustreError: 270218:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8699.906851] LustreError: 270218:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8704.999725] LustreError: 270217:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8705.245327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8705.842968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8711.507846] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8720.047428] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8727.196963] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8734.718540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8735.277974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8736.309515] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 8742.480578] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8750.647949] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8751.727471] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 8755.682030] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8755.686520] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8755.690410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8755.740559] Lustre: Skipped 1 previous similar message [ 8760.006944] Lustre: server umount lustre-OST0000 complete [ 8765.928804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8765.939188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8765.959191] Lustre: Skipped 1 previous similar message [ 8765.971289] Lustre: Skipped 2 previous similar messages [ 8770.632702] Lustre: server umount lustre-MDT0000 complete [ 8771.040679] LustreError: 270218:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8771.056806] LustreError: 270218:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8774.153957] LustreError: 270197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784052523 with bad export cookie 9058973047436775 [ 8774.157927] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8774.163065] LustreError: 270197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8774.459587] Lustre: server umount lustre-MDT0001 complete [ 8785.684115] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 8788.844207] Key type lgssc unregistered [ 8789.282513] LNet: 273804:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8789.298901] LNetError: 273804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8789.330902] LNet: Removed LNI 192.168.204.107@tcp [ 8790.218943] Key type .llcrypt unregistered [ 8790.221528] Key type ._llcrypt unregistered [ 8808.527916] Key type ._llcrypt registered [ 8808.530497] Key type .llcrypt registered [ 8808.634441] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 8809.564225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8809.702858] alg: No test for adler32 (adler32-zlib) [ 8810.802317] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 8811.041359] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 8812.728556] Key type lgssc registered [ 8813.942659] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8823.953987] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8823.997107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8839.647751] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8844.768084] LustreError: 274997:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.107@tcp: failed processing log, type 4: rc = -110 [ 8871.455649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8879.007794] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8889.438800] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8892.552913] Lustre: server umount lustre-OST0000 complete [ 8911.938461] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 8915.267989] Key type lgssc unregistered [ 8915.594733] LNet: 276627:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8915.606334] LNetError: 276627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8915.628420] LNet: Removed LNI 192.168.204.107@tcp [ 8916.702788] Key type .llcrypt unregistered [ 8916.706295] Key type ._llcrypt unregistered [ 8963.456604] Key type ._llcrypt registered [ 8963.461882] Key type .llcrypt registered [ 8963.612801] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 8964.890483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8964.926065] alg: No test for adler32 (adler32-zlib) [ 8966.055882] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 8966.346813] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 8968.087382] Key type lgssc registered [ 8969.414713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8983.223289] loop0: detected capacity change from 0 to 204800000 [ 8983.429696] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8983.441550] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8983.842746] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8985.008255] loop0: detected capacity change from 0 to 204800000 [ 8985.301887] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8985.325032] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8985.730554] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8986.819890] loop0: detected capacity change from 0 to 204800000 [ 8987.082220] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8987.121268] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8987.316229] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8988.475962] loop0: detected capacity change from 0 to 204800000 [ 8988.633797] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8988.646874] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8988.782443] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8989.906731] loop0: detected capacity change from 0 to 204800000 [ 8989.943707] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8994.064484] loop0: detected capacity change from 0 to 204800000 [ 8994.115187] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8998.368550] loop0: detected capacity change from 0 to 204800000 [ 8998.412323] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9002.964579] loop0: detected capacity change from 0 to 204800000 [ 9003.014853] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9007.535962] loop0: detected capacity change from 0 to 204800000 [ 9007.667960] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9007.733132] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9007.833801] Lustre: 280046:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9007.843604] Lustre: 280046:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 9007.854348] Lustre: 280046:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9007.866246] Lustre: 280046:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9007.878729] Lustre: 280046:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9007.891307] Lustre: 280046:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 9007.902262] Lustre: 280046:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 9007.920240] Lustre: 280046:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 9007.932906] Lustre: 280046:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 9007.952547] Lustre: 280046:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 9007.972634] Lustre: 280046:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 9007.988237] Lustre: 280046:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 9007.999554] Lustre: 280046:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 9008.014592] Lustre: 280046:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 9014.451399] Lustre: server umount MGS complete [ 9015.677686] loop0: detected capacity change from 0 to 204800000 [ 9015.819090] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9016.673431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9016.787964] Lustre: lustre-MDT0000: Aborting client recovery [ 9017.906273] loop1: detected capacity change from 0 to 204800000 [ 9018.068114] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9018.184817] Lustre: 280479:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9018.198045] Lustre: 280479:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9018.208756] Lustre: 280479:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 9018.219470] Lustre: 280479:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 9018.233260] Lustre: 280479:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 9018.245947] Lustre: 280479:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9018.260339] Lustre: 280479:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9018.296295] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9018.438212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9018.473364] Lustre: lustre-MDT0001: Aborting client recovery [ 9019.436401] loop2: detected capacity change from 0 to 204800000 [ 9019.630702] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9019.685058] Lustre: 280558:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9019.696240] Lustre: 280558:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9019.720723] Lustre: 280558:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9019.727702] Lustre: 280558:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9019.737660] Lustre: 280558:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9019.744464] Lustre: 280558:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9019.759218] Lustre: 280558:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9019.883747] Lustre: Found index 0 for lustre-OST0000, updating log [ 9019.916364] Lustre: 280577:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9019.958979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9019.977274] Lustre: lustre-OST0000: Aborting recovery [ 9021.088363] loop3: detected capacity change from 0 to 204800000 [ 9021.266544] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9021.378372] Lustre: 280641:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9021.396608] Lustre: 280641:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9021.412065] Lustre: 280641:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9021.435322] Lustre: 280641:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9021.454601] Lustre: 280641:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9021.486311] Lustre: 280641:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9021.505204] Lustre: 280641:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9021.542766] Lustre: Found index 1 for lustre-OST0001, updating log [ 9021.598311] Lustre: 280648:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9021.744421] Lustre: lustre-OST0001: Aborting recovery [ 9025.833709] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9025.848253] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9025.895229] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 9025.907492] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 9025.920149] LustreError: 280423:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 9027.878387] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 9027.888723] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 9031.285066] LustreError: 280389:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 1619us: rc = -2 [ 9031.822271] LustreError: 280386:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 2128us: rc = -2 [ 9031.879497] LustreError: 280386:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 15 previous similar messages [ 9048.547566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9048.555606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9048.571406] Lustre: Skipped 2 previous similar messages [ 9048.585523] Lustre: Skipped 3 previous similar messages [ 9051.858524] Lustre: server umount lustre-MDT0000 complete [ 9053.119202] LustreError: 280371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784052802 with bad export cookie 14152473801288072506 [ 9053.138883] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9053.143028] LustreError: 280371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9053.665444] LustreError: 280387:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9053.669108] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9053.675391] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9053.698919] LustreError: 280387:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9053.724076] Lustre: Skipped 2 previous similar messages [ 9058.785642] LustreError: 280386:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9058.787171] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9058.806563] Lustre: Skipped 2 previous similar messages [ 9059.669086] Lustre: server umount lustre-MDT0001 complete [ 9075.168659] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9075.370049] Lustre: server umount lustre-OST0000 complete [ 9082.752286] Lustre: server umount lustre-OST0001 complete [ 9094.426226] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 14:14:00 (1784052840) [ 9131.975532] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 9143.773855] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 9193.189981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9193.510826] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9193.546286] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9193.663597] Lustre: lustre-MDT0000: new disk, initializing [ 9193.815150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9193.818583] Lustre: Skipped 1 previous similar message [ 9193.853179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9199.230454] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9211.762819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9211.943364] Lustre: 284721:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9212.058500] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9212.063272] Lustre: Skipped 1 previous similar message [ 9212.147845] Lustre: lustre-MDT0001: new disk, initializing [ 9212.264134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9212.325494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9212.342173] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9218.478416] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9229.582360] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9236.648257] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9244.539818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9244.802594] Lustre: lustre-OST0000: new disk, initializing [ 9244.807655] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9244.816728] Lustre: 286689:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9244.929180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9252.133924] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9252.923828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9252.932198] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9253.101780] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9262.218470] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9268.164194] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9282.754941] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9289.241046] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9294.300195] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 9340.387774] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9340.395922] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9340.404574] LustreError: Skipped 1 previous similar message [ 9340.411623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9340.423923] Lustre: Skipped 1 previous similar message [ 9340.469417] Lustre: Skipped 1 previous similar message [ 9345.510603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9345.515165] Lustre: Skipped 1 previous similar message [ 9352.159502] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9352.619765] Lustre: server umount lustre-OST0000 complete [ 9360.881201] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9360.886753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9360.891264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9360.891272] Lustre: Skipped 2 previous similar messages [ 9363.961736] Lustre: server umount lustre-MDT0000 complete [ 9365.984659] LustreError: 287566:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9366.013266] LustreError: 287566:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9368.994945] LustreError: 285516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784053118 with bad export cookie 14152473801288075208 [ 9368.998903] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9369.008566] LustreError: 285516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9369.486335] Lustre: server umount lustre-MDT0001 complete [ 9380.517497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9386.034364] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9391.206747] Lustre: server umount MGS complete [ 9407.113138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9407.803691] LustreError: 290611:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9407.837792] LustreError: 290624:0:(qmt_pool.c:1645:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9407.945589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9413.090464] LustreError: 290612:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9414.710291] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9423.329126] LustreError: 290611:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9423.347720] LustreError: 290611:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9426.949659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9432.774889] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9442.638512] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9450.199155] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9460.045411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9460.561388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9460.566713] Lustre: Skipped 1 previous similar message [ 9467.782889] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9477.269665] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9512.931903] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9512.946877] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9512.955372] Lustre: Skipped 1 previous similar message [ 9512.960427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9512.969338] Lustre: Skipped 1 previous similar message [ 9515.489385] Lustre: server umount lustre-OST0000 complete [ 9524.193331] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9524.212030] Lustre: Skipped 2 previous similar messages [ 9526.434425] Lustre: server umount lustre-MDT0000 complete [ 9529.314468] LustreError: 291341:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9529.333763] LustreError: 291341:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9530.960593] LustreError: 290591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784053280 with bad export cookie 14152473801288077035 [ 9530.965937] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9530.990454] LustreError: 290591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9531.426839] Lustre: server umount lustre-MDT0001 complete [ 9544.202275] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 9548.133596] Key type lgssc unregistered [ 9548.557680] LNet: 294557:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9548.573407] LNetError: 294557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9548.613120] LNet: Removed LNI 192.168.204.107@tcp [ 9549.721036] Key type .llcrypt unregistered [ 9549.724215] Key type ._llcrypt unregistered [ 9593.026502] Key type ._llcrypt registered [ 9593.030080] Key type .llcrypt registered [ 9593.177286] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 9610.426061] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 9611.870299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9611.933086] alg: No test for adler32 (adler32-zlib) [ 9613.113844] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [ 9613.430062] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 9615.151161] Key type lgssc registered [ 9616.403577] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9658.166871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9658.186532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9659.374853] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9659.400169] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9659.486950] Lustre: lustre-MDT0000: new disk, initializing [ 9659.581678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9659.600353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9664.061921] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9675.861719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9676.000907] Lustre: 298515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9676.053110] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9676.056606] Lustre: Skipped 1 previous similar message [ 9676.164518] Lustre: lustre-MDT0001: new disk, initializing [ 9676.357460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9676.423627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9676.455673] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9680.831479] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9694.068856] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9701.854864] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9710.590191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9710.964342] Lustre: lustre-OST0000: new disk, initializing [ 9710.973599] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9710.989197] Lustre: 300487:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9711.131439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9718.026718] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9720.911471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9720.928961] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9721.051903] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9728.885304] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9736.519568] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9749.463534] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9755.265156] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9760.655672] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 9808.352648] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9808.361417] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9808.378922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9808.388770] Lustre: Skipped 1 previous similar message [ 9811.446605] Lustre: server umount lustre-OST0000 complete [ 9818.593242] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9818.595206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9818.611050] Lustre: Skipped 2 previous similar messages [ 9818.619669] Lustre: Skipped 1 previous similar message [ 9823.727888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9823.736705] Lustre: Skipped 1 previous similar message [ 9827.809533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9827.828296] Lustre: Skipped 1 previous similar message [ 9830.367131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9830.608341] Lustre: server umount lustre-MDT0000 complete [ 9835.007374] LustreError: 298510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784053584 with bad export cookie 14409573273814533628 [ 9835.016157] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9835.020670] LustreError: 298510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9835.430265] Lustre: server umount lustre-MDT0001 complete [ 9845.350398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9851.359666] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9857.340218] Lustre: server umount MGS complete [ 9871.115371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9871.839606] LustreError: 304403:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9871.867072] LustreError: 304403:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9871.894916] LustreError: 304416:0:(qmt_pool.c:1645:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9872.043426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9877.399560] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9877.472753] LustreError: 304404:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9882.608098] LustreError: 304403:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9887.272843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9887.688078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9892.123878] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9901.495446] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9911.015150] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9920.486079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9920.794762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9928.887510] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9937.526372] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9973.219018] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9973.242711] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9973.258950] LustreError: Skipped 2 previous similar messages [ 9973.261124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9975.145370] Lustre: server umount lustre-OST0000 complete [ 9979.873630] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9979.880082] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9979.889642] Lustre: Skipped 1 previous similar message [ 9983.465857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9983.473134] Lustre: Skipped 3 previous similar messages [ 9985.607686] Lustre: server umount lustre-MDT0000 complete [ 9988.576853] LustreError: 305693:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9988.613045] LustreError: 305693:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9989.596171] LustreError: 304383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784053738 with bad export cookie 14409573273814535469 [ 9989.602779] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9989.608978] LustreError: 304383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9990.122587] Lustre: server umount lustre-MDT0001 complete [10002.460769] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [10006.268861] Key type lgssc unregistered [10006.849333] LNet: 308353:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10006.860400] LNetError: 308353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10006.887244] LNet: Removed LNI 192.168.204.107@tcp [10007.994666] Key type .llcrypt unregistered [10007.997771] Key type ._llcrypt unregistered [10040.156366] Key type ._llcrypt registered [10040.162158] Key type .llcrypt registered [10040.294966] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [10054.904991] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [10056.286586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10056.374093] alg: No test for adler32 (adler32-zlib) [10057.469067] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [10057.704994] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [10059.415192] Key type lgssc registered [10060.860918] Lustre: Echo OBD driver; http://www.lustre.org/ [10089.817591] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [10105.576325] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10105.589516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10106.893375] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10106.919975] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10106.993729] Lustre: lustre-MDT0000: new disk, initializing [10107.062953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10107.078074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10113.164707] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10125.546553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10125.851192] Lustre: lustre-OST0000: new disk, initializing [10125.863651] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10125.876571] Lustre: Skipped 1 previous similar message [10125.891462] Lustre: 312405:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10126.027120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10133.240273] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10133.632378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10133.643542] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10133.672474] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11237.347245] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11237.361663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11240.852751] Lustre: server umount lustre-MDT0000 complete [11245.526885] LustreError: 311591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784054994 with bad export cookie 2506604041526410394 [11245.545396] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11245.553803] LustreError: 311591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11246.027461] Lustre: server umount lustre-OST0000 complete [11277.493931] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [11291.541233] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [11335.024927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11335.434780] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11335.464590] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11335.541213] Lustre: lustre-MDT0000: new disk, initializing [11335.616664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11335.644345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11340.304863] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11351.877353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11352.000424] Lustre: 316777:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11352.049840] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11352.054877] Lustre: Skipped 1 previous similar message [11352.203091] Lustre: lustre-MDT0001: new disk, initializing [11352.316129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11352.343338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11352.348754] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11357.590160] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11368.347581] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11375.863752] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11382.612612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11383.023966] Lustre: lustre-OST0000: new disk, initializing [11383.036830] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11383.068499] Lustre: 318748:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11383.224618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11391.113612] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11393.118583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11393.140578] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11393.246744] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11405.947808] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11413.504976] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11414.050957] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11419.919968] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11420.153358] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11424.225536] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11424.229623] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11424.240701] LustreError: Skipped 1 previous similar message [11424.242350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11424.265540] Lustre: Skipped 1 previous similar message [11429.663725] Lustre: server umount lustre-OST0000 complete [11434.464741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11434.479257] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11434.494455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11434.515418] Lustre: Skipped 1 previous similar message [11434.539714] Lustre: Skipped 2 previous similar messages [11439.607633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11439.619630] Lustre: Skipped 1 previous similar message [11444.711863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11444.734675] Lustre: Skipped 1 previous similar message [11448.287253] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11448.702822] Lustre: server umount lustre-MDT0000 complete [11449.832126] LustreError: 316790:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11449.855345] LustreError: 316790:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11453.929759] LustreError: 317963:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11453.975623] LustreError: 317963:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11455.469343] LustreError: 316770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784055204 with bad export cookie 2506604041529351220 [11455.505044] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11455.527923] LustreError: 316770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11456.432365] Lustre: server umount lustre-MDT0001 complete [11463.645418] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [11466.267171] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 14:53:32 (1784055212) [11476.256661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11476.659052] LustreError: 321064:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11476.745110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11482.081276] LustreError: 321065:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11482.353771] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11486.182450] LustreError: 321064:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11491.914592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11492.317245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11497.677132] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11509.224746] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11518.999877] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11530.520044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11530.884389] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11537.560381] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11546.133884] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11554.728613] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11554.960252] Lustre: lustre-OST0001: new disk, initializing [11554.963811] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11554.969447] Lustre: 323902:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11555.104752] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11562.781930] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11564.129286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11564.151665] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11564.189770] LustreError: 324331:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11565.222048] LustreError: 324331:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11565.241609] LustreError: 324331:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [11566.304985] LustreError: 324336:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11566.338164] LustreError: 324336:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [11568.353668] LustreError: 324331:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11568.359903] LustreError: 324331:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [11572.454463] LustreError: 324331:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11572.462183] LustreError: 324331:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [11573.650881] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11573.962170] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11580.651196] LustreError: 324331:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11580.664705] LustreError: 324331:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 15 previous similar messages [11581.238992] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff98ca4a1aa800.ost_server_uuid 50 [11583.461925] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff98ca4a1aa800.ost_server_uuid in FULL state after 0 sec [11589.950436] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff98ca4a1aa800.ost_server_uuid 50 [11592.194726] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff98ca4a1aa800.ost_server_uuid in IDLE state after 0 sec [11595.134331] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11609.541712] Lustre: lustre-OST0001: Not available for connect from 192.168.204.7@tcp (stopping) [11610.595787] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11610.612508] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11610.640723] Lustre: Skipped 1 previous similar message [11615.355425] Lustre: server umount lustre-OST0001 complete [11615.721715] LustreError: 324332:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11615.749236] LustreError: 324332:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11623.393517] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11623.402547] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11625.956369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11625.970543] Lustre: Skipped 6 previous similar messages [11628.669264] Lustre: server umount lustre-OST0000 complete [11635.681354] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11635.701458] LustreError: Skipped 1 previous similar message [11635.704547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11635.717049] Lustre: Skipped 1 previous similar message [11639.663137] Lustre: server umount lustre-MDT0000 complete [11641.316171] LustreError: 321064:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11641.356487] LustreError: 321064:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [11644.515600] LustreError: 321045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784055393 with bad export cookie 2506604041529352102 [11644.517135] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11644.521496] LustreError: 321045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11644.789864] Lustre: server umount lustre-MDT0001 complete [11662.363977] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [11666.563230] Key type lgssc unregistered [11666.938259] LNet: 326613:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11666.954206] LNetError: 326613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11668.010175] LNet: Removed LNI 192.168.204.107@tcp [11669.534940] Key type .llcrypt unregistered [11669.542043] Key type ._llcrypt unregistered [11700.718707] Key type ._llcrypt registered [11700.721453] Key type .llcrypt registered [11700.833270] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [11702.099168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11702.463735] alg: No test for adler32 (adler32-zlib) [11703.749655] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [11704.130655] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [11705.799221] Key type lgssc registered [11707.290424] Lustre: Echo OBD driver; http://www.lustre.org/ [11716.417579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11716.459265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11718.151162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11723.930148] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11728.363449] LustreError: 328035:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11728.410918] LustreError: 328035:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11733.489987] LustreError: 328034:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11733.628407] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11738.598037] LustreError: 328035:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11741.718719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11742.160328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11747.594513] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11758.357557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11758.858089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11762.980794] LustreError: 329717:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11763.015692] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [11765.004943] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11767.096330] LustreError: 329715:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11767.140954] LustreError: 329715:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11776.112169] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11777.534518] LustreError: 330166:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11777.556500] LustreError: 330166:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11788.843622] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11789.051453] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11794.429730] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [11797.856545] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11806.784900] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11815.289553] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [11815.533491] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11932.653744] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11932.655702] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11932.657428] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11932.663976] LustreError: Skipped 1 previous similar message [11932.692311] Lustre: Skipped 1 previous similar message [11936.079647] Lustre: server umount lustre-OST0001 complete [11937.763345] LustreError: 329716:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11937.812134] LustreError: 329716:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11946.481553] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11946.492563] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11946.504940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11946.512595] Lustre: Skipped 1 previous similar message [11947.502148] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11947.531573] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11947.572289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11950.736975] Lustre: server umount lustre-OST0000 complete [11957.218110] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11957.233075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11957.254073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11957.257016] Lustre: Skipped 2 previous similar messages [11962.461374] Lustre: server umount lustre-MDT0000 complete [11967.343332] LustreError: 331720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784055716 with bad export cookie 9452162818629190137 [11967.354659] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11967.358384] LustreError: 331720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11968.044716] Lustre: server umount lustre-MDT0001 complete [11984.586634] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [11990.036556] Key type lgssc unregistered [11990.258615] LNet: 332994:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11990.275218] LNetError: 332994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11990.298649] LNet: Removed LNI 192.168.204.107@tcp [11991.218795] Key type .llcrypt unregistered [11991.222862] Key type ._llcrypt unregistered [12050.578942] Key type ._llcrypt registered [12050.581942] Key type .llcrypt registered [12050.688111] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [12051.818996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12051.959924] alg: No test for adler32 (adler32-zlib) [12053.021132] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [12053.294689] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [12055.007183] Key type lgssc registered [12056.221371] Lustre: Echo OBD driver; http://www.lustre.org/ [12067.168067] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12067.197421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12068.736826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12073.925681] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12078.060890] LustreError: 335286:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12078.091211] LustreError: 335286:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12083.113336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12083.174993] LustreError: 335287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12083.470133] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12087.629569] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12090.636675] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12098.478287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12098.936690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12105.983289] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12108.135218] LustreError: 336780:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12113.280081] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [12113.303590] LustreError: 337477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12115.267505] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12115.609104] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12121.123107] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [12123.251959] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12134.037111] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12141.465068] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [12155.360491] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12155.375203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12155.397557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12156.898088] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12156.914234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12156.931602] Lustre: Skipped 1 previous similar message [12156.952708] Lustre: Skipped 1 previous similar message [12161.674199] Lustre: server umount lustre-MDT0000 complete [12162.029844] LustreError: 336339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12162.061126] LustreError: 336339:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12166.430541] LustreError: 335267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784055915 with bad export cookie 16047151048868809661 [12166.440894] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12166.454419] LustreError: 335267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12167.117930] Lustre: server umount lustre-MDT0001 complete [12182.485734] Lustre: server umount lustre-OST0000 complete [12183.327288] Lustre: 334550:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784055916/real 1784055916] req@ffff93e545b57b80 x1870718096703744/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784055932 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12183.371473] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12183.399724] Lustre: Skipped 1 previous similar message [12187.435254] Lustre: server umount lustre-OST0001 complete [12207.979988] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [12221.900636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12222.602216] LustreError: 340559:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12222.695035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12228.536537] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12239.304319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12239.652481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12245.255244] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12248.673521] Lustre: 341699:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [12256.146570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12256.656977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12263.858935] LustreError: 342377:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12263.891686] LustreError: 342377:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12264.523784] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12266.936406] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [12273.656794] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12279.310279] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [12281.672755] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12290.663412] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12294.764203] Lustre: 343572:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [12301.280518] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12301.302082] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12301.331591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12304.878141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12306.885468] Lustre: server umount lustre-MDT0000 complete [12309.992226] LustreError: 340555:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12310.031928] LustreError: 340555:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [12311.547644] LustreError: 340541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784056060 with bad export cookie 16047151048868811362 [12311.549374] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12311.556140] LustreError: 340541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [12311.845860] Lustre: server umount lustre-MDT0001 complete [12326.683034] Lustre: server umount lustre-OST0000 complete [12331.491980] Lustre: 334549:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784056064/real 1784056064] req@ffff93e45ed9c380 x1870718096757632/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784056080 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12331.571431] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12331.599204] Lustre: Skipped 3 previous similar messages [12332.514706] Lustre: 334551:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784056065/real 1784056065] req@ffff93e56f198380 x1870718096757888/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784056081 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12334.465120] Lustre: server umount lustre-OST0001 complete [12339.706460] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [12341.802215] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [12344.241936] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 15:08:11 (1784056091) [12372.291992] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [12383.385209] loop0: detected capacity change from 0 to 1099511627776 [12384.212030] SGI XFS with ACLs, security attributes, quota, no debug enabled [12384.233616] *** VALIDATE xfs *** [12384.248904] XFS (loop0): Mounting V5 Filesystem [12384.294249] XFS (loop0): Ending clean mount [12385.998821] loop1: detected capacity change from 0 to 18691697672192 [12525.931943] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [12538.630566] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [12582.135857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12582.529909] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12582.603683] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12582.746507] Lustre: lustre-MDT0000: new disk, initializing [12582.872869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12582.882303] Lustre: Skipped 1 previous similar message [12582.899732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12588.015393] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12600.385100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12600.499846] Lustre: 348714:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12600.593755] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12600.603326] Lustre: Skipped 1 previous similar message [12600.698458] Lustre: lustre-MDT0001: new disk, initializing [12600.848860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12600.869648] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12606.003594] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12617.935980] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12625.258074] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12633.192232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12633.636897] Lustre: lustre-OST0000: new disk, initializing [12633.640957] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12633.649568] Lustre: 350684:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12634.852338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12634.864815] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12635.017298] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12640.790835] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12651.907573] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12657.754779] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12658.058356] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12663.556122] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12663.845684] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12670.953850] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12670.958380] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12670.993085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12671.008235] Lustre: Skipped 4 previous similar messages [12672.509141] Lustre: server umount lustre-OST0000 complete [12677.602147] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12677.616234] LustreError: Skipped 1 previous similar message [12681.184372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12681.191976] Lustre: Skipped 2 previous similar messages [12683.134863] Lustre: server umount lustre-MDT0000 complete [12686.317598] LustreError: 348722:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12686.341120] LustreError: 348722:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12687.288650] LustreError: 350683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784056436 with bad export cookie 16047151048868812909 [12687.296299] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12687.308270] LustreError: 350683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12687.698412] Lustre: server umount lustre-MDT0001 complete [12697.145370] XFS (loop0): Unmounting Filesystem [12702.988536] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 15:14:10 (1784056450) [12711.347915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12711.949537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12711.957730] Lustre: Skipped 2 previous similar messages [12717.646741] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12727.446921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12733.299207] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12741.465984] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12748.065637] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12756.867975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12764.725273] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12773.542896] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12795.364217] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12795.385518] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12795.418158] Lustre: Skipped 3 previous similar messages [12795.426076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12799.612370] Lustre: server umount lustre-OST0000 complete [12804.576708] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12804.583269] LustreError: Skipped 1 previous similar message [12810.044860] Lustre: server umount lustre-MDT0000 complete [12813.829395] LustreError: 353226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784056563 with bad export cookie 16047151048868813798 [12813.852469] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12813.855718] LustreError: 353226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12814.284106] Lustre: server umount lustre-MDT0001 complete [12824.942197] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [12827.930425] Key type lgssc unregistered [12828.280994] LNet: 356899:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12828.293214] LNetError: 356899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12828.325148] LNet: Removed LNI 192.168.204.107@tcp [12829.405996] Key type .llcrypt unregistered [12829.435587] Key type ._llcrypt unregistered [12857.569336] Key type ._llcrypt registered [12857.573270] Key type .llcrypt registered [12857.661683] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [12858.741353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12858.866711] alg: No test for adler32 (adler32-zlib) [12859.963180] Lustre: Lustre: Build Version: 2.17.54_162_gf04e30c [12860.202760] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [12861.879235] Key type lgssc registered [12863.322584] Lustre: Echo OBD driver; http://www.lustre.org/ [12871.635982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12871.663026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12873.180522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12878.194885] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12883.437716] LustreError: 358468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12883.451371] LustreError: 358468:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12887.837737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12888.278227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12893.346890] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12901.962680] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12907.725694] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12913.914046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12914.084387] LustreError: 360288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12914.252765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12919.845073] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12927.701121] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12934.582320] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.7@tcp (stopping) [12934.632024] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12934.636412] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12939.243133] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12939.244691] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12939.252277] Lustre: Skipped 1 previous similar message [12941.016285] Lustre: server umount lustre-MDT0001 complete [12944.359955] LustreError: 359196:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12944.389073] LustreError: 359196:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12949.476088] LustreError: 358462:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12949.500369] LustreError: 358462:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12952.788467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12952.903639] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12952.908607] Lustre: Found index 1 for lustre-MDT0001, updating log [12952.960948] Lustre: 358455:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12953.330256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12958.102865] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12958.720301] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12961.803015] LustreError: 362069:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12961.811163] LustreError: 362069:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12961.836667] LustreError: 358459:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12961.941764] LustreError: 362072:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.107@tcp: cfg command failed: rc = -17 [12961.955294] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [12961.955294] [13265.888411] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13265.890584] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13265.901696] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13265.921203] Lustre: Skipped 1 previous similar message [13265.930141] Lustre: Skipped 1 previous similar message [13270.508313] Lustre: server umount lustre-MDT0001 complete [13271.016206] LustreError: 358463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13271.045927] LustreError: 358463:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13280.654904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13280.748943] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13280.756721] Lustre: Found index 1 for lustre-MDT0001, updating log [13280.785431] Lustre: 358456:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13281.062779] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13284.837662] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [13286.379052] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13289.480618] LustreError: 370592:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13289.491658] LustreError: 370592:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13289.499210] LustreError: 358459:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13289.505414] LustreError: 358459:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [13289.586542] LustreError: 370595:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [13289.600095] LustreError: 370595:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [13289.610992] LustreError: 370595:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.107@tcp: cfg command failed: rc = -17 [13289.627481] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.204.107@tcp [13289.627481] [13589.953891] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.7@tcp (stopping) [13589.963218] Lustre: Skipped 2 previous similar messages [13593.569320] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13593.593000] Lustre: Skipped 1 previous similar message [13593.599774] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13595.036349] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.7@tcp (stopping) [13595.047756] Lustre: Skipped 2 previous similar messages [13603.815358] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13603.820516] Lustre: Skipped 4 previous similar messages [13604.319147] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13604.482895] Lustre: server umount lustre-MDT0001 complete [13605.276056] LustreError: 358463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13605.294858] LustreError: 358463:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13616.018743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13616.182256] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13616.198239] Lustre: Found index 1 for lustre-MDT0001, updating log [13616.259833] Lustre: 358456:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13616.599280] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13621.315437] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [13621.744351] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13623.790723] LustreError: 379134:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13623.804144] LustreError: 379134:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13623.821469] LustreError: 358459:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13623.831845] LustreError: 358459:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [13623.910460] LustreError: 379141:0:(lod_lov.c:211:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [13623.929895] LustreError: 379141:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.107@tcp: cfg command failed: rc = -114 [13623.940414] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [13623.940414] [13944.292438] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13944.299066] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13944.299735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13944.306704] Lustre: Skipped 2 previous similar messages [13949.012894] Lustre: server umount lustre-MDT0000 complete [13949.408626] LustreError: 358462:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13949.429034] LustreError: 358462:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [13952.904586] LustreError: 358446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784057702 with bad export cookie 16782502640790695186 [13952.914975] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13952.925051] LustreError: 358446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13954.543394] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13967.327169] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13967.702157] Lustre: server umount lustre-MDT0001 complete [13972.425878] Lustre: server umount lustre-OST0000 complete [13981.653988] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [13993.272483] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [14033.949496] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 15:36:21 (1784057781) [14041.696528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14042.099762] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14042.157849] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14042.244936] Lustre: lustre-MDT0000: new disk, initializing [14042.392041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14042.413328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14047.748429] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14059.722349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14059.830533] Lustre: 389750:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14059.906214] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14059.912643] Lustre: Skipped 1 previous similar message [14059.991571] Lustre: lustre-MDT0001: new disk, initializing [14060.103451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14060.134406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14060.141331] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14064.930865] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14074.859751] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14081.525854] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14089.363603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14089.749756] Lustre: lustre-OST0000: new disk, initializing [14089.755330] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14089.762548] Lustre: 391720:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14089.836557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14096.560650] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14097.033066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14097.049579] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14097.187041] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14106.141264] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14114.790687] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [14114.794242] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14114.824528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14114.829071] Lustre: Skipped 6 previous similar messages [14115.807605] Lustre: server umount lustre-MDT0000 complete [14117.857450] LustreError: 391221:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14117.886583] LustreError: 391221:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [14119.912707] LustreError: 389742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784057869 with bad export cookie 16782502640790697139 [14119.917055] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14119.920411] LustreError: 389742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14120.355358] Lustre: server umount lustre-MDT0001 complete [14134.222691] Lustre: server umount lustre-OST0000 complete [14164.522544] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [14175.452863] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [14211.812372] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13884 sec ========= 15:39:18 (1784057958) [14214.553200] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:39:21 (1784057961) === [14218.809916] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:39:25 (1784057965) === [14269.143284] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [14272.094842] Key type lgssc unregistered [14272.453539] LNet: 398089:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14272.464230] LNetError: 398089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14272.484480] LNet: Removed LNI 192.168.204.107@tcp [14273.746596] Key type .llcrypt unregistered [14273.749702] Key type ._llcrypt unregistered