[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 489664115 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003353] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.007755] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010010] pid_max: default: 32768 minimum: 301 [ 0.011122] LSM: Security Framework initializing [ 0.012057] Yama: becoming mindful. [ 0.013033] SELinux: Initializing. [ 0.014068] *** VALIDATE selinux *** [ 0.019643] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.023141] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024106] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025110] *** VALIDATE tmpfs *** [ 0.026441] *** VALIDATE proc *** [ 0.028110] *** VALIDATE cgroup *** [ 0.029029] *** VALIDATE cgroup2 *** [ 0.030250] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032006] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034023] Spectre V2 : User space: Vulnerable [ 0.035006] Speculative Store Bypass: Vulnerable [ 0.038495] debug: unmapping init [mem 0xffffffff94259000-0xffffffff94260fff] [ 0.040170] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041703] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042020] ... version: 2 [ 0.043010] ... bit width: 48 [ 0.044010] ... generic registers: 4 [ 0.045011] ... value mask: 0000ffffffffffff [ 0.046011] ... max period: 00007fffffffffff [ 0.047010] ... fixed-purpose events: 3 [ 0.048010] ... event mask: 000000070000000f [ 0.049267] rcu: Hierarchical SRCU implementation. [ 0.051429] smp: Bringing up secondary CPUs ... [ 0.052553] x86: Booting SMP configuration: [ 0.053022] .... node #0, CPUs: #1 #2 #3 [ 0.058078] smp: Brought up 1 node, 4 CPUs [ 0.060014] smpboot: Max logical packages: 1 [ 0.061012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.146019] node 0 deferred pages initialised in 81ms [ 0.149142] devtmpfs: initialized [ 0.150210] x86/mm: Memory block size: 128MB [ 0.154406] gcov: version magic: 0x41383552 [ 0.157432] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.160069] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.163392] pinctrl core: initialized pinctrl subsystem [ 0.165148] [ 0.165726] ************************************************************* [ 0.168012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171010] ** ** [ 0.173009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.175010] ** ** [ 0.178011] ** This means that this kernel is built to expose internal ** [ 0.180008] ** IOMMU data structures, which may compromise security on ** [ 0.183011] ** your system. ** [ 0.184010] ** ** [ 0.187015] ** If you see this message and you are not debugging the ** [ 0.189010] ** kernel, report this immediately to your vendor! ** [ 0.191012] ** ** [ 0.194011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.196022] ************************************************************* [ 0.199662] NET: Registered protocol family 16 [ 0.201422] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.204056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.207054] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.211112] cpuidle: using governor menu [ 0.212765] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.214419] PCI: Using configuration type 1 for base access [ 0.216131] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.227117] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.228045] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.230105] cryptd: max_cpu_qlen set to 1000 [ 0.234116] ACPI: Added _OSI(Module Device) [ 0.236014] ACPI: Added _OSI(Processor Device) [ 0.238012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.239020] ACPI: Added _OSI(Processor Aggregator Device) [ 0.246464] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.252395] ACPI: Interpreter enabled [ 0.254171] ACPI: PM: (supports S0 S3 S4 S5) [ 0.257027] ACPI: Using IOAPIC for interrupt routing [ 0.260094] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.264395] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.275647] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.277033] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.280017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.283066] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.288276] acpiphp: Slot [2] registered [ 0.289101] acpiphp: Slot [5] registered [ 0.291099] acpiphp: Slot [6] registered [ 0.292084] acpiphp: Slot [7] registered [ 0.293115] acpiphp: Slot [8] registered [ 0.295123] acpiphp: Slot [9] registered [ 0.298148] acpiphp: Slot [10] registered [ 0.299105] acpiphp: Slot [3] registered [ 0.301094] acpiphp: Slot [4] registered [ 0.302089] acpiphp: Slot [11] registered [ 0.304091] acpiphp: Slot [12] registered [ 0.305104] acpiphp: Slot [13] registered [ 0.307068] acpiphp: Slot [14] registered [ 0.309096] acpiphp: Slot [15] registered [ 0.311272] acpiphp: Slot [16] registered [ 0.313106] acpiphp: Slot [17] registered [ 0.315096] acpiphp: Slot [18] registered [ 0.317165] acpiphp: Slot [19] registered [ 0.318231] acpiphp: Slot [20] registered [ 0.320145] acpiphp: Slot [21] registered [ 0.322174] acpiphp: Slot [22] registered [ 0.325122] acpiphp: Slot [23] registered [ 0.328173] acpiphp: Slot [24] registered [ 0.330149] acpiphp: Slot [25] registered [ 0.331090] acpiphp: Slot [26] registered [ 0.333126] acpiphp: Slot [27] registered [ 0.336120] acpiphp: Slot [28] registered [ 0.337111] acpiphp: Slot [29] registered [ 0.339170] acpiphp: Slot [30] registered [ 0.342127] acpiphp: Slot [31] registered [ 0.344063] PCI host bridge to bus 0000:00 [ 0.347023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.351025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.354032] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.357023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.361022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.363037] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.365143] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.367936] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.370085] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.379735] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.385058] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.387014] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.389023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.391012] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.393561] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.397100] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.399040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.402646] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.407013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.420015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.426015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.432868] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.446013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.455014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.477017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.491606] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.503014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.516015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.536015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.547615] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.558012] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.569018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.590013] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.603028] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.615014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.626015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.649017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.661720] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.672016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.683015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.710018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.722317] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.737000] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.757015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.807016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.831283] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.835386] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.836535] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.837457] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.839187] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.844031] iommu: Default domain type: Passthrough [ 0.845366] SCSI subsystem initialized [ 0.846119] ACPI: bus type USB registered [ 0.847191] usbcore: registered new interface driver usbfs [ 0.849060] usbcore: registered new interface driver hub [ 0.850132] usbcore: registered new device driver usb [ 0.852140] pps_core: LinuxPPS API ver. 1 registered [ 0.853008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.856094] PTP clock support registered [ 0.858227] EDAC MC: Ver: 3.0.0 [ 0.860013] PCI: Using ACPI for IRQ routing [ 0.861820] NetLabel: Initializing [ 0.863009] NetLabel: domain hash size = 128 [ 0.864008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.865070] NetLabel: unlabeled traffic allowed by default [ 0.867106] vgaarb: loaded [ 0.869394] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.870128] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.874648] clocksource: Switched to clocksource kvm-clock [ 0.979767] VFS: Disk quotas dquot_6.6.0 [ 0.981030] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.982954] *** VALIDATE ramfs *** [ 0.984209] *** VALIDATE hugetlbfs *** [ 0.985645] pnp: PnP ACPI init [ 0.987835] pnp: PnP ACPI: found 6 devices [ 1.009783] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.013339] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.016050] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.018567] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.020881] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.023534] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.026582] NET: Registered protocol family 2 [ 1.029098] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.033342] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.036293] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.041597] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.045228] TCP: Hash tables configured (established 65536 bind 65536) [ 1.047888] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.050626] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.053360] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.055748] NET: Registered protocol family 1 [ 1.057930] RPC: Registered named UNIX socket transport module. [ 1.059586] RPC: Registered udp transport module. [ 1.060845] RPC: Registered tcp transport module. [ 1.062209] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.064521] NET: Registered protocol family 44 [ 1.065989] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.067914] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.069968] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.072360] PCI: CLS 0 bytes, default 64 [ 1.074857] Unpacking initramfs... [ 2.512424] debug: unmapping init [mem 0xffff8c5bfcc54000-0xffff8c5bfffbffff] [ 2.516685] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.519052] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.522114] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.016985] Initialise system trusted keyrings [ 3.018762] Key type blacklist registered [ 3.020384] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.035754] zbud: loaded [ 3.038972] *** VALIDATE nfs *** [ 3.040669] *** VALIDATE nfs4 *** [ 3.042866] pstore: using deflate compression [ 3.047306] Platform Keyring initialized [ 3.153225] NET: Registered protocol family 38 [ 3.154869] Key type asymmetric registered [ 3.156255] Asymmetric key parser 'x509' registered [ 3.158307] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.161568] io scheduler mq-deadline registered [ 3.163435] io scheduler kyber registered [ 3.165226] io scheduler bfq registered [ 3.167208] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.170740] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.173625] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.175983] ACPI: Power Button [PWRF] [ 3.182217] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.189214] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.200458] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.207340] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.220223] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.249102] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.276442] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.280757] Non-volatile memory driver v1.3 [ 3.282303] Linux agpgart interface v0.103 [ 3.310386] virtio_blk virtio1: [vda] 146152 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.313504] vda: detected capacity change from 0 to 74829824 [ 3.331406] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.334595] vdb: detected capacity change from 0 to 1073741824 [ 3.350889] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.353992] vdc: detected capacity change from 0 to 2621440000 [ 3.369689] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.373104] vdd: detected capacity change from 0 to 2621440000 [ 3.388772] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.391101] vde: detected capacity change from 0 to 4294967296 [ 3.403748] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.406025] vdf: detected capacity change from 0 to 4294967296 [ 3.414433] libphy: Fixed MDIO Bus: probed [ 3.419786] usbcore: registered new interface driver usbserial_generic [ 3.421985] usbserial: USB Serial support registered for generic [ 3.424428] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.428787] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.430778] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.432850] mousedev: PS/2 mouse device common for all mice [ 3.436428] rtc_cmos 00:05: RTC can wake from S4 [ 3.439826] rtc_cmos 00:05: registered as rtc0 [ 3.441593] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.444242] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.444293] intel_pstate: CPU model not supported [ 3.450902] hid: raw HID events driver (C) Jiri Kosina [ 3.451558] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.453083] usbcore: registered new interface driver usbhid [ 3.459223] usbhid: USB HID core driver [ 3.460400] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.460669] drop_monitor: Initializing network drop monitor service [ 3.464972] Initializing XFRM netlink socket [ 3.466691] NET: Registered protocol family 10 [ 3.469843] Segment Routing with IPv6 [ 3.471336] NET: Registered protocol family 17 [ 3.473929] mpls_gso: MPLS GSO support [ 3.479986] RAS: Correctable Errors collector initialized. [ 3.482349] AVX version of gcm_enc/dec engaged. [ 3.483956] AES CTR mode by8 optimization enabled [ 3.560294] sched_clock: Marking stable (3560025566, 0)->(4521684858, -961659292) [ 3.565122] registered taskstats version 1 [ 3.568113] Loading compiled-in X.509 certificates [ 3.571772] zswap: loaded using pool lzo/zbud [ 3.603539] Key type big_key registered [ 3.618657] Key type encrypted registered [ 3.620863] ima: No TPM chip found, activating TPM-bypass! [ 3.623488] ima: Allocated hash algorithm: sha1 [ 3.625694] ima: No architecture policies found [ 3.627598] evm: Initialising EVM extended attributes: [ 3.629186] evm: security.selinux [ 3.630206] evm: security.ima [ 3.631138] evm: security.capability [ 3.632183] evm: HMAC attrs: 0x1 [ 3.635280] rtc_cmos 00:05: setting system clock to 2026-08-24 13:52:42 UTC (1787579562) [ 3.642754] debug: unmapping init [mem 0xffffffff95203000-0xffffffff953fffff] [ 3.646387] debug: unmapping init [mem 0xffffffff93f82000-0xffffffff94258fff] [ 3.655077] Write protecting the kernel read-only data: 28672k [ 3.658505] debug: unmapping init [mem 0xffffffff92603000-0xffffffff927fffff] [ 3.661507] debug: unmapping init [mem 0xffffffff92f14000-0xffffffff92ffffff] [ 3.692430] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.702632] systemd[1]: Detected virtualization kvm. [ 3.705443] systemd[1]: Detected architecture x86-64. [ 3.708175] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.735645] systemd[1]: No hostname configured. [ 3.737755] systemd[1]: Set hostname to . [ 3.739957] random: systemd: uninitialized urandom read (16 bytes read) [ 3.742309] systemd[1]: Initializing machine ID from random generator. [ 3.873456] random: systemd: uninitialized urandom read (16 bytes read) [ 3.877323] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.885408] random: systemd: uninitialized urandom read (16 bytes read) [ 3.889172] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.896115] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.548391] device-mapper: uevent: version 1.0.3 [ 4.552089] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.015323] virtio_net virtio0 ens2: renamed from eth0 [ 6.090351] random: fast init done [ 6.207043] scsi host0: ata_piix [ 6.291952] scsi host1: ata_piix [ 6.293658] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.296668] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.144259] random: crng init done [ 11.148108] random: 7 urandom warning(s) missed due to ratelimiting [ 12.440813] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 14.452482] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.470314] printk: systemd: 26 output lines suppressed due to ratelimiting [ 18.294523] SELinux: Disabled at runtime. [ 18.448562] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.484487] systemd[1]: Detected virtualization kvm. [ 18.488680] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 20.120818] systemd[1]: initrd-switch-root.service: Succeeded. [ 20.132050] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 20.143379] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 20.155517] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 20.161401] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 20.176859] systemd[1]: Starting Journal Service... Starting Journal Service... [ 20.202314] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Created slice User and Session Slice. [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 20.646054] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Slices. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 21.619063] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 22.545401] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 22.654221] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 23.153194] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 23.186957] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit)[ 28.424191] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 29.075528] NFS: Registering the id_resolver key type [ 29.079243] Key type id_resolver registered [ 29.084231] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ 36.637602] hrtimer: interrupt took 12047675 ns [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg654-server login: [ 91.951744] libcfs: loading out-of-tree module taints kernel. [ 92.054627] Key type ._llcrypt registered [ 92.063589] Key type .llcrypt registered [ 92.232483] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [ 111.423932] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 112.904914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 112.930651] alg: No test for adler32 (adler32-zlib) [ 114.899404] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 115.769502] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [ 117.616854] Key type lgssc registered [ 120.219839] Lustre: Echo OBD driver; http://www.lustre.org/ [ 137.292117] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 179.195749] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 191.965926] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 191.990827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 193.247485] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 193.287638] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 193.377258] Lustre: lustre-MDT0000: new disk, initializing [ 193.458207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 193.471048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 197.446783] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 210.029315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 210.126845] Lustre: 6513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 210.151938] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 210.160218] Lustre: Skipped 1 previous similar message [ 210.226922] Lustre: lustre-MDT0001: new disk, initializing [ 210.283351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 210.310501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 210.324247] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 214.727538] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 219.308342] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 228.637349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 228.794848] Lustre: lustre-OST0000: new disk, initializing [ 228.797219] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 228.801175] Lustre: 8448:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 228.840942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 232.676413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 232.691143] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 232.807083] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 236.755263] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 253.882444] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 254.047602] Lustre: lustre-OST0001: new disk, initializing [ 254.055308] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 254.068849] Lustre: 9522:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 254.182085] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 260.327697] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 263.770078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 263.784706] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 263.856244] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 273.011567] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 281.241039] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 288.840503] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing check_logdir /tmp/testlogs/ [ 293.612193] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing yml_node [ 297.871123] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 300.552988] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 303.229148] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 304.745699] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 24 09:57:42 EDT 2026 [ 321.494093] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 322.905436] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 338.400950] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 338.414620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 338.437261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 340.963034] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 340.964253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 340.986305] Lustre: Skipped 1 previous similar message [ 341.002827] Lustre: Skipped 3 previous similar messages [ 344.796396] Lustre: server umount lustre-MDT0000 complete [ 348.433169] LustreError: 9521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787579907 with bad export cookie 7045906010347540845 [ 348.439453] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 348.443512] LustreError: 9521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 351.201886] LustreError: 6519:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 351.203291] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 351.204493] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 351.237539] LustreError: 6519:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 351.253799] Lustre: Skipped 2 previous similar messages [ 353.249127] LustreError: 9531:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 353.249656] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 353.266418] LustreError: 9531:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 353.272907] Lustre: Skipped 2 previous similar messages [ 354.943751] Lustre: server umount lustre-MDT0001 complete [ 359.206286] Lustre: server umount lustre-OST0000 complete [ 362.812985] Lustre: server umount lustre-OST0001 complete [ 370.191357] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [ 380.049742] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 418.541564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 418.730801] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 418.777895] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 418.844677] Lustre: lustre-MDT0000: new disk, initializing [ 418.906930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 418.937129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 423.972060] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 435.706472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 435.795971] Lustre: 15075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 435.838994] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 435.841396] Lustre: Skipped 1 previous similar message [ 435.917950] Lustre: lustre-MDT0001: new disk, initializing [ 436.053753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 436.105031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 436.125783] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 440.387821] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 451.074516] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 457.755644] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 465.512850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 465.747488] Lustre: lustre-OST0000: new disk, initializing [ 465.752785] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 465.760367] Lustre: 17046:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 465.853197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 466.890279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 466.908106] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 467.079185] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 473.132822] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 482.929160] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 487.654430] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 487.872840] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 492.298111] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 492.508310] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 497.641836] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 497.643323] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 497.649132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 497.659756] Lustre: Skipped 1 previous similar message [ 497.694297] LustreError: Skipped 1 previous similar message [ 500.918837] Lustre: server umount lustre-OST0000 complete [ 507.875791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 507.876768] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 507.888460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 507.888468] Lustre: Skipped 1 previous similar message [ 511.620662] Lustre: server umount lustre-MDT0000 complete [ 512.993637] LustreError: 15962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 513.017795] LustreError: 15962:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 515.870211] LustreError: 15067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580074 with bad export cookie 7045906010347542378 [ 515.877900] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 515.884397] LustreError: 15067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 516.293346] Lustre: server umount lustre-MDT0001 complete [ 521.174331] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 10:01:18 (1787580078) [ 528.294792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 528.816777] LustreError: 19324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 528.893430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 533.767468] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 533.987444] LustreError: 19325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 543.128044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 548.202682] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 555.366511] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 560.444609] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 567.357120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 574.136821] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 582.006258] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 585.310520] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 594.406790] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 594.429834] Lustre: Skipped 1 previous similar message [ 594.434159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 594.446243] Lustre: Skipped 1 previous similar message [ 599.339810] Lustre: server umount lustre-MDT0000 complete [ 599.528936] LustreError: 19324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 599.558092] LustreError: 19324:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 603.212079] LustreError: 19305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580162 with bad export cookie 7045906010347543253 [ 603.227067] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 603.229813] LustreError: 19305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 603.555947] Lustre: server umount lustre-MDT0001 complete [ 617.965309] Lustre: server umount lustre-OST0000 complete [ 639.649413] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 650.024463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 650.451840] LustreError: 23705:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 650.510545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 650.515903] Lustre: Skipped 2 previous similar messages [ 655.752861] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 666.652467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 672.492629] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 675.578444] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 682.598939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 690.279134] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 699.902402] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 700.033426] Lustre: lustre-OST0001: new disk, initializing [ 700.036528] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 700.042163] Lustre: 26021:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 701.450425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 701.465838] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 701.581264] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 706.129845] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 716.650921] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 720.364987] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 733.669521] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 733.687960] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 733.705058] Lustre: Skipped 1 previous similar message [ 733.714691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 739.695563] Lustre: server umount lustre-MDT0000 complete [ 742.383923] LustreError: 26645:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 742.408288] LustreError: 26645:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 743.604952] LustreError: 26788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580302 with bad export cookie 7045906010347544107 [ 743.616726] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 743.624840] LustreError: 26788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 744.163613] Lustre: server umount lustre-MDT0001 complete [ 757.439318] Lustre: server umount lustre-OST0000 complete [ 771.731087] Lustre: server umount lustre-OST0001 complete [ 787.814367] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 798.337293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 798.831513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 798.838747] Lustre: Skipped 3 previous similar messages [ 803.579766] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 808.491300] LustreError: 29043:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 808.510272] LustreError: 29043:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 812.166920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 816.901789] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 820.100956] Lustre: 30190:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 827.497554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 834.937770] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 845.021617] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 852.758255] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 861.112482] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 865.011266] Lustre: 32068:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 873.964090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 873.970982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 873.992395] Lustre: Skipped 3 previous similar messages [ 874.002306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 874.011281] Lustre: Skipped 4 previous similar messages [ 878.108368] Lustre: server umount lustre-MDT0000 complete [ 881.900699] LustreError: 32072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580440 with bad export cookie 7045906010347545402 [ 881.905627] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 881.911262] LustreError: 32072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 882.303415] Lustre: server umount lustre-MDT0001 complete [ 895.338393] Lustre: server umount lustre-OST0000 complete [ 910.298414] Lustre: server umount lustre-OST0001 complete [ 920.482575] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 10:07:57 (1787580477) [ 948.284749] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 958.408612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 958.852670] LustreError: 34875:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 958.875729] LustreError: 34875:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 963.739541] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 972.563323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 977.804650] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 980.801298] Lustre: 36024:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 987.726661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 994.841993] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 1004.295844] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1011.366493] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 1019.412979] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1023.398912] Lustre: 37902:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1029.602536] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1029.618434] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1029.629783] Lustre: Skipped 3 previous similar messages [ 1029.635271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1029.639510] Lustre: Skipped 4 previous similar messages [ 1035.079866] Lustre: server umount lustre-MDT0000 complete [ 1038.708212] LustreError: 34855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580597 with bad export cookie 7045906010347546508 [ 1038.722190] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1039.146139] Lustre: server umount lustre-MDT0001 complete [ 1052.328736] Lustre: server umount lustre-OST0000 complete [ 1056.223865] Lustre: 3646:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787580599/real 1787580599] req@ffff8c5b47b38e00 x1874413144587904/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787580615 ref 1 fl Rpc:EXNQr/200/ffffffff rc -5/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1066.788854] Lustre: server umount lustre-OST0001 complete [ 1076.158939] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 10:10:33 (1787580633) [ 1092.573107] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 1102.682595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1103.237367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1103.244175] Lustre: Skipped 7 previous similar messages [ 1107.830549] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 1116.717087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1121.902521] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 1124.862979] Lustre: 41194:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1131.668389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1139.226564] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 1147.841786] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1155.014950] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 1163.545662] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1169.030086] Lustre: 43071:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1170.164612] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1170.179867] Lustre: Skipped 1 previous similar message [ 1175.016685] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1175.047750] Lustre: Skipped 6 previous similar messages [ 1180.472396] Lustre: server umount lustre-MDT0000 complete [ 1184.800739] LustreError: 40027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580743 with bad export cookie 7045906010347547607 [ 1184.810737] LustreError: 40027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1184.811416] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1191.389027] Lustre: server umount lustre-MDT0001 complete [ 1201.318204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1215.771765] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 1219.694991] LustreError: 41558:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1219.697770] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1219.722070] LustreError: 41558:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 31 previous similar messages [ 1224.624246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1230.025419] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 1230.336089] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1230.352637] Lustre: Skipped 1 previous similar message [ 1238.268907] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1239.783275] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1245.296362] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1247.144896] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1255.922474] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1258.247822] Lustre: server umount lustre-MDT0000 complete [ 1261.709366] LustreError: 40027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580820 with bad export cookie 7045906010347548706 [ 1261.715969] LustreError: 40027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1261.721523] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1261.989595] Lustre: server umount lustre-MDT0001 complete [ 1275.454779] Lustre: server umount lustre-OST0000 complete [ 1288.494507] Lustre: server umount lustre-OST0001 complete [ 1297.827193] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 10:14:15 (1787580855) [ 1314.920200] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 1324.215890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1329.161524] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 1338.659706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1344.578862] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 1347.660281] Lustre: 48833:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1355.705360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1362.064246] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 1371.601986] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1378.152916] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 1385.995840] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1391.282400] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1405.711243] Lustre: server umount lustre-MDT0000 complete [ 1405.924255] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1409.194539] LustreError: 47663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580968 with bad export cookie 7045906010347549532 [ 1409.195900] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1409.207267] LustreError: 47663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 1409.549795] Lustre: server umount lustre-MDT0001 complete [ 1424.134694] Lustre: server umount lustre-OST0000 complete [ 1428.433033] Lustre: 3649:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787580971/real 1787580971] req@ffff8c5c488b4380 x1874413144735488/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787580987 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1428.840235] Lustre: server umount lustre-OST0001 complete [ 1440.448890] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 10:16:37 (1787580997) [ 1443.283933] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1445.365201] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 10:16:42 (1787581002) [ 1458.165061] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [ 1468.384914] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 1495.884903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1496.331342] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1496.379766] Lustre: lustre-MDT0000: new disk, initializing [ 1496.430420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1500.481612] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 1511.048892] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1518.296720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1518.640925] Lustre: lustre-OST0000: new disk, initializing [ 1518.645477] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1518.650858] Lustre: Skipped 1 previous similar message [ 1518.658125] Lustre: 54905:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1520.609427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1520.621100] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1520.667552] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1524.542881] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 1534.992637] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1537.422381] Lustre: DEBUG MARKER: create test files [ 1576.929595] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1576.937256] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1576.948558] Lustre: Skipped 11 previous similar messages [ 1576.953624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1576.963425] Lustre: Skipped 18 previous similar messages [ 1586.147603] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1586.496749] Lustre: server umount lustre-OST0000 complete [ 1591.649687] Lustre: server umount lustre-MDT0000 complete [ 1605.848492] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [ 1609.622294] Key type lgssc unregistered [ 1610.276511] LNet: 56543:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1610.299132] LNetError: 56543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1611.368171] LNet: Removed LNI 192.168.206.154@tcp [ 1612.487166] Key type .llcrypt unregistered [ 1612.489191] Key type ._llcrypt unregistered [ 1646.936927] Key type ._llcrypt registered [ 1646.939864] Key type .llcrypt registered [ 1647.241108] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 1649.101584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1649.152856] alg: No test for adler32 (adler32-zlib) [ 1650.371024] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 1650.824361] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [ 1652.578630] Key type lgssc registered [ 1655.001853] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1665.956148] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1665.975277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1667.419484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1673.398552] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 1681.554852] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1688.640171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1688.945870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1693.054449] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:101 to 0x240000400:129) [ 1693.226074] Lustre: 59281:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 511 < left 544, rollback = 9 [ 1693.247888] Lustre: 59281:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/4, destroy: 0/0/0 [ 1693.273084] Lustre: 59281:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1693.291889] Lustre: 59281:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1693.314862] Lustre: 59281:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1693.332380] Lustre: 59281:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1696.190652] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 1704.548715] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1710.574120] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1716.070405] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1728.993936] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1729.008894] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1729.037885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1731.852802] Lustre: server umount lustre-OST0000 complete [ 1737.562690] Lustre: server umount lustre-MDT0000 complete [ 1750.923383] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [ 1753.765940] Key type lgssc unregistered [ 1754.076470] LNet: 60897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1754.097932] LNetError: 60897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1754.124636] LNet: Removed LNI 192.168.206.154@tcp [ 1754.862969] Key type .llcrypt unregistered [ 1754.866121] Key type ._llcrypt unregistered [ 1789.344746] Key type ._llcrypt registered [ 1789.348937] Key type .llcrypt registered [ 1789.516878] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 1790.993553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1791.073737] alg: No test for adler32 (adler32-zlib) [ 1792.291862] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 1792.614274] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [ 1794.354086] Key type lgssc registered [ 1795.508347] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1803.496080] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1803.520414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1804.996568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1809.707459] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 1818.068097] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1825.358157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1825.676049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1829.809815] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1829.908601] Lustre: 63734:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1829.920383] Lustre: 63734:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1829.939904] Lustre: 63734:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1829.954035] Lustre: 63734:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1829.963088] Lustre: 63734:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1829.971894] Lustre: 63734:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1832.154826] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 1840.262395] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1846.031224] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1855.472968] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1855.481494] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1855.491710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1860.490640] Lustre: server umount lustre-OST0000 complete [ 1865.384575] Lustre: server umount lustre-MDT0000 complete [ 1878.768830] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [ 1882.114994] Key type lgssc unregistered [ 1882.434226] LNet: 65294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1882.443572] LNetError: 65294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1883.497418] LNet: Removed LNI 192.168.206.154@tcp [ 1884.644359] Key type .llcrypt unregistered [ 1884.648630] Key type ._llcrypt unregistered [ 1906.415185] Key type ._llcrypt registered [ 1906.417023] Key type .llcrypt registered [ 1906.531893] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [ 1923.964243] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 1925.250670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1925.261691] alg: No test for adler32 (adler32-zlib) [ 1926.501428] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 1926.955289] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [ 1928.744678] Key type lgssc registered [ 1930.348623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1969.668643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1969.730438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1971.012917] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1971.075830] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1971.261084] Lustre: lustre-MDT0000: new disk, initializing [ 1971.460578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1971.491111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1977.492931] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 1991.329446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1991.511762] Lustre: 68451:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1991.597135] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1991.606300] Lustre: Skipped 1 previous similar message [ 1991.762337] Lustre: lustre-MDT0001: new disk, initializing [ 1991.924432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1991.988029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1992.001285] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1997.714739] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 2008.384052] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2014.405453] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2021.319741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2021.633232] Lustre: lustre-OST0000: new disk, initializing [ 2021.636940] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2021.644629] Lustre: 70419:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2021.756347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2028.704914] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 2029.675723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2029.697085] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2029.951224] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2039.760922] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2045.029417] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2045.179250] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2049.915687] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2050.188624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2055.649921] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2055.656629] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2055.668607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2055.711128] Lustre: Skipped 1 previous similar message [ 2060.774152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2060.778070] Lustre: Skipped 2 previous similar messages [ 2065.903490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2065.908922] Lustre: Skipped 1 previous similar message [ 2066.400132] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2066.535905] Lustre: server umount lustre-OST0000 complete [ 2071.012041] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2071.034871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2071.043912] Lustre: Skipped 1 previous similar message [ 2076.131506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2076.148147] Lustre: Skipped 1 previous similar message [ 2084.320274] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2084.453389] LustreError: 71959:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x5:0x0].0x0 (00000000b96384f9) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2084.680264] Lustre: server umount lustre-MDT0000 complete [ 2088.405428] LustreError: 68443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787581647 with bad export cookie 4950153450901140387 [ 2088.424091] LustreError: 68443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2088.425123] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2088.871244] Lustre: server umount lustre-MDT0001 complete [ 2098.986678] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 10:27:35 (1787581655) [ 2111.151856] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 2151.045832] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [ 2161.708633] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 2200.814060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2201.010819] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2201.048917] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2201.143107] Lustre: lustre-MDT0000: new disk, initializing [ 2201.242087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2201.253219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2206.548342] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 2217.389925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2217.579361] Lustre: 76443:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2217.642171] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2217.651367] Lustre: Skipped 1 previous similar message [ 2217.711954] Lustre: lustre-MDT0001: new disk, initializing [ 2217.788667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2217.825106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2217.835725] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2222.462538] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 2232.221156] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2238.037775] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2245.996212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2246.305991] Lustre: lustre-OST0000: new disk, initializing [ 2246.309603] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2246.315616] Lustre: 78411:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2246.446921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2253.036068] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 2253.907625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2253.924043] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2254.111129] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2263.505784] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2268.605992] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2268.883709] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2273.834885] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2274.081929] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2279.906071] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2279.907452] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2279.921129] LustreError: Skipped 1 previous similar message [ 2279.928300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2279.930892] Lustre: Skipped 2 previous similar messages [ 2279.950989] Lustre: Skipped 3 previous similar messages [ 2282.515907] Lustre: server umount lustre-OST0000 complete [ 2289.634058] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2289.650074] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2299.363465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2299.376751] Lustre: Skipped 6 previous similar messages [ 2300.896131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2301.332680] Lustre: server umount lustre-MDT0000 complete [ 2304.492483] LustreError: 77659:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2304.522151] LustreError: 77659:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2305.534150] LustreError: 76435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787581864 with bad export cookie 4950153450901141269 [ 2305.553105] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2306.068824] Lustre: server umount lustre-MDT0001 complete [ 2318.393954] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 10:31:14 (1787581874) [ 2326.942832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2327.420817] LustreError: 80827:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2327.492333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2332.644485] LustreError: 80828:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2335.119631] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 2337.768076] LustreError: 80827:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2342.885447] LustreError: 80828:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2346.011022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2351.871747] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 2358.990126] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2365.347014] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2373.167985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2373.572048] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2373.578589] Lustre: Skipped 1 previous similar message [ 2380.469903] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 2388.475572] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2392.400284] Lustre: 83499:0:(genops.c:1774:obd_export_evict_by_uuid()) MGS: evicting 370d2008-203d-42fc-bfde-53535ad24cad at adminstrative request [ 2397.157754] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2397.180136] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2407.400691] Lustre: MGS: Client 370d2008-203d-42fc-bfde-53535ad24cad (at 0@lo) reconnecting [ 2407.413174] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x44b27a5628a2fa87 to 0x44b27a5628a2fd20 [ 2407.437233] Lustre: MGC192.168.206.154@tcp: Connection restored to 0@lo (at 0@lo) [ 2430.212732] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2430.421836] Lustre: lustre-OST0001: new disk, initializing [ 2430.424840] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2430.435873] Lustre: 83892:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2437.187911] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 2438.333537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2438.345747] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2438.472467] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2448.633913] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2453.986857] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2453.996978] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2454.004705] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2454.010832] Lustre: Skipped 2 previous similar messages [ 2458.675556] Lustre: server umount lustre-OST0001 complete [ 2459.118577] LustreError: 82647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2459.146073] LustreError: 82647:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2463.719285] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2463.728033] LustreError: Skipped 1 previous similar message [ 2463.737163] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2468.851713] Lustre: server umount lustre-OST0000 complete [ 2478.712911] Lustre: server umount lustre-MDT0000 complete [ 2479.588321] LustreError: 80822:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2479.610948] LustreError: 80822:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2482.917134] LustreError: 80808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787582041 with bad export cookie 4950153450901142816 [ 2482.921934] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2482.931971] LustreError: 80808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2483.608722] Lustre: server umount lustre-MDT0001 complete [ 2493.426520] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 10:34:10 (1787582050) [ 2495.326650] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2497.331575] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 10:34:14 (1787582054) [ 2499.819752] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2501.996869] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 10:34:19 (1787582059) [ 2503.882739] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2506.009144] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 10:34:23 (1787582063) [ 2507.830642] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2509.638191] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 10:34:27 (1787582067) [ 2511.711382] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2519.644802] loop: module loaded [ 2519.654372] loop0: detected capacity change from 0 to 4294967296 [ 2519.723016] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2521.668429] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2525.124341] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2529.864437] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2559.964880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2560.035663] LustreError: 88021:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2560.050653] LustreError: 88021:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2560.062488] LustreError: 88021:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2560.075090] LustreError: 88021:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2582.100788] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [ 2594.592330] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 2634.057137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2634.362721] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2634.401359] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2634.490390] Lustre: lustre-MDT0000: new disk, initializing [ 2634.559741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2634.565975] Lustre: Skipped 1 previous similar message [ 2634.580102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2639.237667] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 2650.942676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2651.075186] Lustre: 90888:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2651.391144] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2656.758793] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 2667.206441] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2673.140328] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2680.983607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2681.462799] Lustre: 92856:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2688.655360] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 2690.174649] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2690.350363] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2699.922369] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2705.086703] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2705.310922] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2710.404853] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2710.721726] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2716.139175] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2716.152711] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2716.153028] Lustre: Skipped 4 previous similar messages [ 2716.153105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2716.153109] Lustre: Skipped 6 previous similar messages [ 2718.939747] Lustre: server umount lustre-OST0000 complete [ 2729.137902] Lustre: server umount lustre-MDT0000 complete [ 2731.491231] LustreError: 90897:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2731.515691] LustreError: 90897:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2732.480994] LustreError: 90883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787582291 with bad export cookie 4950153450901143439 [ 2732.485136] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2732.489038] LustreError: 90883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2732.904651] Lustre: server umount lustre-MDT0001 complete [ 2742.416951] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 10:38:19 (1787582299) [ 2749.338933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2754.262849] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 2763.166585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2763.729196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-60 [ 2763.743219] Lustre: Skipped 3 previous similar messages [ 2768.421574] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 2777.504961] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2783.092823] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2790.241463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2797.313902] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 2805.050881] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2810.053209] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2810.340622] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2817.201484] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2817.462214] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2827.709734] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2828.052615] Lustre: lustre-OST0001: new disk, initializing [ 2828.064930] Lustre: Skipped 2 previous similar messages [ 2828.076208] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2828.096179] Lustre: Skipped 3 previous similar messages [ 2828.111050] Lustre: 98641:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2834.013498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2834.027274] Lustre: Skipped 2 previous similar messages [ 2834.031281] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2834.215578] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2836.339578] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 2847.762795] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2852.139120] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2852.354642] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2856.904671] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2857.162654] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2880.002251] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2910.901315] Lustre: Failing over lustre-MDT0000 [ 2911.209475] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2911.214860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2911.235744] Lustre: Skipped 4 previous similar messages [ 2911.255980] Lustre: Skipped 7 previous similar messages [ 2911.498416] Lustre: server umount lustre-MDT0000 complete [ 2915.826657] LustreError: 95264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2915.858064] LustreError: 95264:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2932.705088] Lustre: 66672:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787582475/real 1787582475] req@0000000048c420e9 x1874415043501952/t0(0) o400->MGC192.168.206.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787582491 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2932.740810] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2937.975576] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2937.980314] LDISKFS-fs (dm-0): recovery complete [ 2937.991122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2942.948519] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x44b27a5628a3c7a2 [ 2942.967607] Lustre: MGC192.168.206.154@tcp: Connection restored to 0@lo (at 0@lo) [ 2943.288372] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2946.534421] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2946.535911] Lustre: 95266:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2947.907639] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 2948.601151] Lustre: 95270:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2948.623244] Lustre: 95270:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 1 previous similar message [ 2948.650610] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2948.796906] LustreError: 101500:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2949.131035] LustreError: 101500:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2949.461826] LustreError: 101500:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2949.470288] LustreError: 101500:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 1 previous similar message [ 2949.784141] LustreError: 101500:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2949.802258] LustreError: 101500:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 1 previous similar message [ 2949.809882] Lustre: 101500:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2949.817706] Lustre: 101500:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 7 previous similar messages [ 2950.466096] LustreError: 101500:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2950.484091] LustreError: 101500:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 2 previous similar messages [ 2950.808117] LustreError: 101500:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2950.814765] LustreError: 101500:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 2 previous similar messages [ 2952.083624] Lustre: 101500:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2952.113739] Lustre: 101500:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 6 previous similar messages [ 2952.762524] LustreError: 101500:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2952.774241] LustreError: 101500:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 6 previous similar messages [ 2953.088529] LustreError: 101500:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2953.092142] LustreError: 101500:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 6 previous similar messages [ 2956.280932] Lustre: 101500:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2956.291455] Lustre: 101500:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 12 previous similar messages [ 2956.936885] LustreError: 101500:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2956.948614] LustreError: 101500:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 12 previous similar messages [ 2957.280129] LustreError: 101500:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2957.290921] LustreError: 101500:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 12 previous similar messages [ 2964.308703] Lustre: 101500:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2964.323059] Lustre: 101500:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 24 previous similar messages [ 2964.970918] LustreError: 101500:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2964.999117] LustreError: 101500:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 24 previous similar messages [ 2965.336145] LustreError: 101500:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2965.351602] LustreError: 101500:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 24 previous similar messages [ 2980.407550] Lustre: 101500:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2980.422353] Lustre: 101500:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 49 previous similar messages [ 2981.069302] LustreError: 101500:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2981.079351] LustreError: 101500:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 49 previous similar messages [ 2981.400311] LustreError: 101500:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2981.416087] LustreError: 101500:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 49 previous similar messages [ 3006.312129] Lustre: 101500:0:(ldlm_lib.c:2419:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 3006.335394] Lustre: 101500:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3006.351834] Lustre: 101500:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 3ce40c9b-eb9c-4bce-a338-c39e5ac14cb0@192.168.206.54@tcp [ 3006.379316] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 3006.407965] Lustre: 101500:0:(ldlm_lib.c:1939:abort_req_replay_queue()) @@@ aborted: req@00000000d1be2635 x1874415025462272/t0(8589934780) o35->3ce40c9b-eb9c-4bce-a338-c39e5ac14cb0@192.168.206.54@tcp:30/0 lens 392/0 e 0 to 0 dl 1787582575 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 3006.428588] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3006.445445] Lustre: lustre-MDT0000: Denying connection for new client 3ce40c9b-eb9c-4bce-a338-c39e5ac14cb0 (at 192.168.206.54@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 3006.467483] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3006.529560] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 3006.539814] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3006.559601] Lustre: Skipped 2 previous similar messages [ 3006.610901] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:80 to 0x280000401:577) [ 3006.612444] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:48 to 0x2c0000401:545) [ 3027.937899] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3027.949043] LustreError: Skipped 1 previous similar message [ 3027.962708] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3028.001797] Lustre: Skipped 1 previous similar message [ 3031.890130] Lustre: server umount lustre-OST0000 complete [ 3041.313926] Lustre: server umount lustre-OST0001 complete [ 3051.472445] Lustre: server umount lustre-MDT0000 complete [ 3055.319266] LustreError: 95250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787582614 with bad export cookie 4950153450901194658 [ 3055.321073] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3055.330448] LustreError: 95250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3055.678738] Lustre: server umount lustre-MDT0001 complete [ 3064.841893] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 10:43:42 (1787582622) [ 3072.249936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3072.263527] Lustre: *** cfs_fail_loc=197, val=0*** [ 3087.905605] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3093.025177] LustreError: 103483:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.154@tcp: failed processing log, type 4: rc = -110 [ 3098.193977] Lustre: *** cfs_fail_loc=197, val=0*** [ 3098.197845] Lustre: Skipped 65 previous similar messages [ 3118.752325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3118.762382] Lustre: Skipped 3 previous similar messages [ 3125.839891] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 3133.833613] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3135.844979] Lustre: server umount lustre-OST0000 complete [ 3146.125166] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 10:45:03 (1787582703) [ 3167.467988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3167.987842] LustreError: 105102:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3168.014342] LustreError: 105102:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 42 previous similar messages [ 3173.897995] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 3185.391893] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3189.568049] LustreError: 105133:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3189.587332] LustreError: 105133:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3189.595949] LustreError: 105133:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 3190.139412] Lustre: server umount lustre-MDT0000 complete [ 3226.466959] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [ 3236.458529] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 3275.253420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3275.638041] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3275.688512] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3275.764232] Lustre: lustre-MDT0000: new disk, initializing [ 3275.850574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3281.094490] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 3295.322609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3295.445770] Lustre: 109307:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3295.688632] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3301.585603] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 3312.664744] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3319.806476] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3327.187913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3327.582573] Lustre: 111276:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3333.349975] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3334.820698] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 3344.328400] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3351.040245] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3351.403759] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3357.528947] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3357.870225] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3364.329038] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3364.331933] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3364.345138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3364.345145] Lustre: Skipped 12 previous similar messages [ 3364.369347] LustreError: Skipped 2 previous similar messages [ 3364.393937] Lustre: Skipped 6 previous similar messages [ 3366.689652] Lustre: server umount lustre-OST0000 complete [ 3378.767948] Lustre: server umount lustre-MDT0000 complete [ 3383.093746] LustreError: 111275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787582941 with bad export cookie 4950153450901205284 [ 3383.095338] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3383.126140] LustreError: 111275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3383.769661] Lustre: server umount lustre-MDT0001 complete [ 3393.399362] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 10:49:10 (1787582950) [ 3403.548626] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [ 3406.329749] Key type lgssc unregistered [ 3406.657169] LNet: 113770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3406.665418] LNetError: 113770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3406.688259] LNet: Removed LNI 192.168.206.154@tcp [ 3407.584907] Key type .llcrypt unregistered [ 3407.588636] Key type ._llcrypt unregistered [ 3425.067518] Key type ._llcrypt registered [ 3425.069783] Key type .llcrypt registered [ 3425.175948] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [ 3440.875526] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 3441.983023] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3442.058334] alg: No test for adler32 (adler32-zlib) [ 3443.144149] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 3443.449220] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [ 3445.104175] Key type lgssc registered [ 3446.221038] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3485.471432] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3485.506808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3486.767913] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3486.821882] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3486.949170] Lustre: lustre-MDT0000: new disk, initializing [ 3487.037197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3487.067352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3491.522941] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 3501.163690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3501.389590] Lustre: lustre-OST0000: new disk, initializing [ 3501.393365] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3501.397292] Lustre: Skipped 1 previous similar message [ 3501.401637] Lustre: 117945:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3501.477158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3507.829514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3507.837630] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3507.863706] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3508.261943] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 3519.936994] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3520.261317] Lustre: lustre-OST0001: new disk, initializing [ 3520.276884] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3520.291582] Lustre: 118807:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3520.437382] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3526.747941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3526.753633] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3526.797769] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3528.200531] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 3541.989195] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3541.998561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3542.001703] Lustre: Skipped 1 previous similar message [ 3542.012771] Lustre: Skipped 1 previous similar message [ 3546.395729] Lustre: server umount lustre-MDT0000 complete [ 3552.807459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3558.117383] LustreError: 119330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787583116 with bad export cookie 14888540315424602392 [ 3558.130525] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3558.142646] LustreError: 119330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3558.285616] Lustre: server umount lustre-OST0000 complete [ 3561.903268] Lustre: server umount lustre-OST0001 complete [ 3569.956908] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 10:52:07 (1787583127) [ 3590.675731] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 10:52:28 (1787583148) [ 3600.083704] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 3621.806749] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 10:52:59 (1787583179) [ 3661.546298] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [ 3673.472450] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 3709.179975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3709.482597] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3709.560652] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3709.680518] Lustre: lustre-MDT0000: new disk, initializing [ 3709.780273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3709.810353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3713.740525] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 3723.812386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3723.996633] Lustre: 126565:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3724.066949] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3724.070846] Lustre: Skipped 1 previous similar message [ 3724.176823] Lustre: lustre-MDT0001: new disk, initializing [ 3724.278727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3724.315265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3724.325884] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3728.669116] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 3738.547924] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3744.684581] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3752.047191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3752.406729] Lustre: lustre-OST0000: new disk, initializing [ 3752.410790] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3752.416892] Lustre: 128532:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3752.480908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3758.088883] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 3760.723953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3760.734334] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3760.844292] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3768.019361] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3773.212557] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3773.374753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3779.778805] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3780.088366] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3786.736516] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3786.745296] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3786.760529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3786.773690] LustreError: Skipped 1 previous similar message [ 3786.812783] Lustre: Skipped 1 previous similar message [ 3788.842287] Lustre: server umount lustre-OST0000 complete [ 3795.943982] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3795.950619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3795.958831] Lustre: Skipped 1 previous similar message [ 3795.964060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3799.264905] Lustre: server umount lustre-MDT0000 complete [ 3803.506566] LustreError: 128542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787583362 with bad export cookie 14888540315424603617 [ 3803.518032] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3803.525654] LustreError: 128542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3803.794278] Lustre: server umount lustre-MDT0001 complete [ 3813.845827] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 10:56:10 (1787583370) [ 3820.128268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3820.645753] LustreError: 130944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3820.664753] LustreError: 130944:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3820.726068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3825.309723] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 3826.152833] LustreError: 130945:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3830.245321] LustreError: 130944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3834.084677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3838.955696] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 3846.574056] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3852.238510] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3858.893264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3859.116594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3859.124996] Lustre: Skipped 1 previous similar message [ 3865.518721] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 3873.663779] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3878.117017] Lustre: *** cfs_fail_loc=159, val=0*** [ 3878.128118] Lustre: Skipped 1 previous similar message [ 3894.807704] Lustre: lustre-MDT0000: Client 09f18815-fb66-4abe-9ecd-1e6155ef4155 (at 192.168.206.54@tcp) reconnecting [ 3896.684724] Lustre: *** cfs_fail_loc=159, val=0*** [ 3896.686422] Lustre: Skipped 4 previous similar messages [ 3912.231447] Lustre: lustre-MDT0000: Client 09f18815-fb66-4abe-9ecd-1e6155ef4155 (at 192.168.206.54@tcp) reconnecting [ 3921.889639] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3921.892841] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3921.896534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3921.896542] Lustre: Skipped 2 previous similar messages [ 3921.902078] LustreError: Skipped 1 previous similar message [ 3921.935160] Lustre: Skipped 2 previous similar messages [ 3924.371977] Lustre: server umount lustre-OST0000 complete [ 3931.645529] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3931.668097] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3931.690765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3931.702578] Lustre: Skipped 1 previous similar message [ 3941.354570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3941.363557] Lustre: Skipped 5 previous similar messages [ 3943.392812] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3943.717773] Lustre: server umount lustre-MDT0000 complete [ 3946.466852] LustreError: 133543:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3946.497478] LustreError: 133543:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3947.290746] LustreError: 130924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787583506 with bad export cookie 14888540315424604492 [ 3947.297415] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3947.307170] LustreError: 130924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3947.709235] Lustre: server umount lustre-MDT0001 complete [ 3958.653267] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [ 3961.459302] Key type lgssc unregistered [ 3961.814737] LNet: 134720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3961.832971] LNetError: 134720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3961.854633] LNet: Removed LNI 192.168.206.154@tcp [ 3962.972313] Key type .llcrypt unregistered [ 3962.975631] Key type ._llcrypt unregistered [ 3994.682494] Key type ._llcrypt registered [ 3994.684885] Key type .llcrypt registered [ 3994.780788] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 3996.246922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3996.319172] alg: No test for adler32 (adler32-zlib) [ 3997.309566] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 3997.473507] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [ 3999.200870] Key type lgssc registered [ 4000.231672] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4007.246361] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4007.282195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4008.739284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4012.635770] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 4018.146106] LustreError: 136135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4018.169333] LustreError: 136135:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4020.915654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4021.296225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4025.788837] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 4032.802588] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4038.812702] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4046.180714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4046.604565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4052.252480] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 4053.818969] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4059.002743] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4068.779908] Lustre: *** cfs_fail_loc=159, val=0*** [ 4084.249834] Lustre: lustre-MDT0000: Client 3ac68a23-e3c6-4a34-b399-d037a4edca15 (at 192.168.206.54@tcp) reconnecting [ 4086.814734] Lustre: *** cfs_fail_loc=159, val=0*** [ 4086.825740] Lustre: Skipped 1 previous similar message [ 4102.185129] Lustre: lustre-MDT0001: Client 3ac68a23-e3c6-4a34-b399-d037a4edca15 (at 192.168.206.54@tcp) reconnecting [ 4103.747392] Lustre: *** cfs_fail_loc=159, val=0*** [ 4103.750763] Lustre: Skipped 2 previous similar messages [ 4120.103092] Lustre: lustre-MDT0001: Client 3ac68a23-e3c6-4a34-b399-d037a4edca15 (at 192.168.206.54@tcp) reconnecting [ 4135.689564] Lustre: 139204:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4138.659191] Lustre: *** cfs_fail_loc=159, val=0*** [ 4138.664897] Lustre: Skipped 2 previous similar messages [ 4154.411491] Lustre: lustre-MDT0000: Client 981834b5-c0d9-4476-a7b3-7fa48a3cc755 (at 192.168.206.54@tcp) reconnecting [ 4156.067115] Lustre: *** cfs_fail_loc=159, val=0*** [ 4156.071219] Lustre: Skipped 14 previous similar messages [ 4171.324828] Lustre: lustre-MDT0000: Client 981834b5-c0d9-4476-a7b3-7fa48a3cc755 (at 192.168.206.54@tcp) reconnecting [ 4172.873714] Lustre: 139501:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4192.233095] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4192.237832] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4192.248742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4192.286707] Lustre: Skipped 1 previous similar message [ 4197.359055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4197.375077] Lustre: Skipped 2 previous similar messages [ 4198.298289] Lustre: server umount lustre-OST0000 complete [ 4202.466699] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4202.468637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4202.498892] Lustre: Skipped 1 previous similar message [ 4202.512335] Lustre: Skipped 1 previous similar message [ 4207.585828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4207.601318] Lustre: Skipped 1 previous similar message [ 4208.521378] Lustre: server umount lustre-MDT0000 complete [ 4212.611076] LustreError: 136117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787583771 with bad export cookie 17599393270621670970 [ 4212.612205] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4212.635454] LustreError: 136117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4213.027982] Lustre: server umount lustre-MDT0001 complete [ 4224.824343] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [ 4227.651342] Key type lgssc unregistered [ 4227.943726] LNet: 140586:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4227.960258] LNetError: 140586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4227.997740] LNet: Removed LNI 192.168.206.154@tcp [ 4229.065404] Key type .llcrypt unregistered [ 4229.068611] Key type ._llcrypt unregistered [ 4256.468855] Key type ._llcrypt registered [ 4256.473846] Key type .llcrypt registered [ 4256.622715] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 4257.791150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4257.923408] alg: No test for adler32 (adler32-zlib) [ 4259.054878] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 4259.344778] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [ 4261.128167] Key type lgssc registered [ 4262.579936] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4272.078826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4272.147742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4273.631845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4277.905925] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 4283.874861] LustreError: 142010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4283.911934] LustreError: 142010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4287.258096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4287.702263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4292.418266] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 4300.086691] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4306.029769] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4313.602461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4314.013454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4319.164993] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4320.429303] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 4327.834201] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4332.459753] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4336.933493] Lustre: 144593:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4340.219359] Lustre: 144642:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4341.629798] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4342.424732] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4346.599875] Lustre: 144789:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4347.467310] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4348.385708] Lustre: 144887:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4352.996096] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4352.996183] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4353.002923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4353.007117] Lustre: Skipped 1 previous similar message [ 4358.121108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4358.130984] Lustre: Skipped 2 previous similar messages [ 4358.532821] Lustre: server umount lustre-OST0000 complete [ 4363.241714] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4363.249911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4363.252176] Lustre: Skipped 1 previous similar message [ 4363.261316] Lustre: Skipped 1 previous similar message [ 4368.344276] Lustre: server umount lustre-MDT0000 complete [ 4368.371773] LustreError: 142006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4368.386088] LustreError: 142006:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4372.041996] LustreError: 141992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787583930 with bad export cookie 6173732372041775709 [ 4372.048350] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4372.055330] LustreError: 141992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4372.318675] Lustre: server umount lustre-MDT0001 complete [ 4381.984721] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [ 4384.157454] Key type lgssc unregistered [ 4384.405467] LNet: 145950:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4384.412778] LNetError: 145950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4384.428985] LNet: Removed LNI 192.168.206.154@tcp [ 4385.250416] Key type .llcrypt unregistered [ 4385.257521] Key type ._llcrypt unregistered [ 4407.366215] Key type ._llcrypt registered [ 4407.370087] Key type .llcrypt registered [ 4407.476131] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 4408.502267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4408.564138] alg: No test for adler32 (adler32-zlib) [ 4409.585961] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 4409.802704] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [ 4411.448167] Key type lgssc registered [ 4412.514856] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4419.800991] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4419.822343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4421.293364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4425.381782] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 4431.842585] LustreError: 147367:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4431.871896] LustreError: 147367:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4432.793500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4433.180720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4437.288586] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 4444.081475] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4449.272106] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4455.391610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4455.778709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4457.850894] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4461.320216] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 4467.067909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4468.762300] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4473.902689] Lustre: *** cfs_fail_loc=159, val=0*** [ 4489.238341] Lustre: lustre-MDT0000: Client 451d5dd8-8c29-4e18-af34-dea9c5c7bff5 (at 192.168.206.54@tcp) reconnecting [ 4508.132558] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4508.141714] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4508.154483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4508.160309] Lustre: Skipped 1 previous similar message [ 4512.920162] Lustre: server umount lustre-OST0000 complete [ 4518.377747] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4518.382429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4518.392756] Lustre: Skipped 1 previous similar message [ 4518.397983] Lustre: Skipped 1 previous similar message [ 4521.930083] Lustre: server umount lustre-MDT0000 complete [ 4523.489486] LustreError: 148427:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4523.528509] LustreError: 148427:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4525.313294] LustreError: 149979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787584084 with bad export cookie 8799984436618254263 [ 4525.317653] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4525.327455] LustreError: 149979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4525.631852] Lustre: server umount lustre-MDT0001 complete [ 4536.126778] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [ 4538.804896] Key type lgssc unregistered [ 4539.235851] LNet: 151038:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4539.254784] LNetError: 151038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4539.283313] LNet: Removed LNI 192.168.206.154@tcp [ 4540.595070] Key type .llcrypt unregistered [ 4540.602046] Key type ._llcrypt unregistered [ 4567.180203] Key type ._llcrypt registered [ 4567.186297] Key type .llcrypt registered [ 4567.309345] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 4569.012420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4569.056615] alg: No test for adler32 (adler32-zlib) [ 4570.129832] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 4570.415112] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [ 4572.168416] Key type lgssc registered [ 4573.166154] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4581.685249] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4581.733378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4583.463081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4588.359221] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 4593.634446] LustreError: 152455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4593.653792] LustreError: 152455:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4595.684433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4596.061186] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4600.506651] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 4606.978238] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4611.674993] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4617.220095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4617.573052] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4620.656732] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4622.514611] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 4628.863902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4629.082488] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4649.053236] Lustre: lustre-OST0000: evicting 3a565004-528c-49dd-81d8-d9ccfb539b1f (at 192.168.206.54@tcp) by administrative request [ 4649.062603] Lustre: Skipped 1 previous similar message [ 4680.165994] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4680.171528] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4680.177972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4680.190513] Lustre: Skipped 1 previous similar message [ 4680.205380] Lustre: Skipped 1 previous similar message [ 4684.456521] Lustre: server umount lustre-OST0000 complete [ 4688.355958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4688.363370] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4688.386156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4690.402805] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4690.405130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4690.427340] Lustre: Skipped 1 previous similar message [ 4694.443077] Lustre: server umount lustre-MDT0000 complete [ 4695.526757] LustreError: 152455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4695.569194] LustreError: 152455:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4698.367842] LustreError: 153195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787584257 with bad export cookie 15061174401484463110 [ 4698.371501] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4698.380205] LustreError: 153195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4699.028278] Lustre: server umount lustre-MDT0001 complete [ 4712.145617] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [ 4715.796735] Key type lgssc unregistered [ 4716.047164] LNet: 156526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4716.066180] LNetError: 156526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4716.091723] LNet: Removed LNI 192.168.206.154@tcp [ 4716.903508] Key type .llcrypt unregistered [ 4716.907903] Key type ._llcrypt unregistered [ 4804.868628] Key type ._llcrypt registered [ 4804.876938] Key type .llcrypt registered [ 4804.989878] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 4805.968654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4806.085094] alg: No test for adler32 (adler32-zlib) [ 4807.052385] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 4807.283457] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [ 4808.977052] Key type lgssc registered [ 4810.342767] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4817.895932] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4817.920814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4819.346304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4823.976679] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 4829.666287] LustreError: 159021:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4829.691138] LustreError: 159021:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4832.131187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4832.495400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4836.705916] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 4843.836482] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4849.376449] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4855.849542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4856.382511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4857.410472] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 4862.487146] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 4864.566760] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 4869.836652] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4879.841349] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4879.842551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4879.862028] Lustre: Skipped 1 previous similar message [ 4884.969393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4884.985051] Lustre: Skipped 1 previous similar message [ 4885.372379] Lustre: server umount lustre-OST0000 complete [ 4889.057883] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4889.074394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4889.097577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4889.105083] Lustre: Skipped 1 previous similar message [ 4890.086470] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4895.216249] LustreError: 159017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4895.231552] LustreError: 159017:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4895.274434] Lustre: server umount lustre-MDT0000 complete [ 4899.580351] LustreError: 159001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787584458 with bad export cookie 11533355677716885692 [ 4899.594919] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4899.601178] LustreError: 159001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4900.085362] Lustre: server umount lustre-MDT0001 complete [ 4910.834091] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [ 4913.911540] Key type lgssc unregistered [ 4914.177236] LNet: 162564:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4914.186408] LNetError: 162564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4914.199755] LNet: Removed LNI 192.168.206.154@tcp [ 4915.208264] Key type .llcrypt unregistered [ 4915.211073] Key type ._llcrypt unregistered [ 4938.619632] Key type ._llcrypt registered [ 4938.621838] Key type .llcrypt registered [ 4938.745636] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 4939.867190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4939.903753] alg: No test for adler32 (adler32-zlib) [ 4940.895822] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 4941.140494] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [ 4942.784259] Key type lgssc registered [ 4943.936946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4959.578139] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 11:15:16 (1787584516) [ 4976.002239] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [ 4986.067331] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 5019.458697] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5019.487440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5020.778672] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5020.811365] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5020.925151] Lustre: lustre-MDT0000: new disk, initializing [ 5021.012335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5021.038534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5025.209069] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 5035.117263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5035.239610] Lustre: 166829:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5035.296133] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5035.307266] Lustre: Skipped 1 previous similar message [ 5035.443505] Lustre: lustre-MDT0001: new disk, initializing [ 5035.558271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5035.618838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5035.640100] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5040.321265] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 5049.552432] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5055.245176] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5061.344411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5061.636600] Lustre: lustre-OST0000: new disk, initializing [ 5061.645824] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5061.654574] Lustre: 168795:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5061.752890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5067.654516] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 5069.396152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5069.411757] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5069.527904] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5076.854589] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5090.273794] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5090.275674] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5090.286406] LustreError: Skipped 1 previous similar message [ 5090.289411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5090.301364] Lustre: Skipped 1 previous similar message [ 5090.319971] Lustre: Skipped 1 previous similar message [ 5095.402249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5095.408756] Lustre: Skipped 1 previous similar message [ 5100.000502] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5100.239818] Lustre: server umount lustre-OST0000 complete [ 5105.635088] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5105.674866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5105.685786] Lustre: Skipped 1 previous similar message [ 5109.915862] Lustre: server umount lustre-MDT0000 complete [ 5113.508422] LustreError: 166819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787584672 with bad export cookie 3066102834344124968 [ 5113.525690] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5113.525873] LustreError: 166819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5113.959070] Lustre: server umount lustre-MDT0001 complete [ 5124.080987] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [ 5126.807536] Key type lgssc unregistered [ 5127.123292] LNet: 170707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5127.131448] LNetError: 170707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5127.148851] LNet: Removed LNI 192.168.206.154@tcp [ 5128.111093] Key type .llcrypt unregistered [ 5128.114647] Key type ._llcrypt unregistered [ 5152.139446] Key type ._llcrypt registered [ 5152.148124] Key type .llcrypt registered [ 5152.277639] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 5153.248815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5153.281749] alg: No test for adler32 (adler32-zlib) [ 5154.386894] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 5154.609876] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [ 5156.320292] Key type lgssc registered [ 5157.392996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5164.223692] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5164.260677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5165.688241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5169.873829] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 5175.286676] LustreError: 172121:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5175.311231] LustreError: 172121:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5179.031342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5179.549470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5184.549051] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 5192.389613] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5198.982765] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5205.920932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5206.295447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5211.465035] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5212.603266] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 5220.174724] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5228.985671] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5229.004430] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5229.023872] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5229.030968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5229.044956] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5229.918687] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5229.942106] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5229.952277] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5229.958740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5229.967702] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5230.786984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5231.650913] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5231.657866] Lustre: Skipped 1 previous similar message [ 5231.671843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5231.684953] Lustre: Skipped 1 previous similar message [ 5231.690876] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5231.695564] Lustre: Skipped 1 previous similar message [ 5231.701247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5231.710973] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5231.721505] Lustre: Skipped 1 previous similar message [ 5232.689815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5233.565600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5234.336321] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5234.340662] Lustre: Skipped 2 previous similar messages [ 5234.354542] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5234.363435] Lustre: Skipped 2 previous similar messages [ 5234.366674] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5234.370960] Lustre: Skipped 2 previous similar messages [ 5234.374194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5234.387053] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5234.391610] Lustre: Skipped 2 previous similar messages [ 5235.124453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5235.967518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5236.667200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5237.700210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5238.597024] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5238.601428] Lustre: Skipped 4 previous similar messages [ 5238.622588] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5238.634334] Lustre: Skipped 4 previous similar messages [ 5238.639855] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5238.644732] Lustre: Skipped 4 previous similar messages [ 5238.648119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5238.657452] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5238.666801] Lustre: Skipped 4 previous similar messages [ 5239.509393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5240.403655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5241.259678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5242.042382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5242.876893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5243.669072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5244.509939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5245.293871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5246.128817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5247.004249] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5247.008910] Lustre: Skipped 9 previous similar messages [ 5247.033067] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5247.047940] Lustre: Skipped 9 previous similar messages [ 5247.054284] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5247.065236] Lustre: Skipped 9 previous similar messages [ 5247.073803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5247.088794] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5247.095051] Lustre: Skipped 9 previous similar messages [ 5247.917314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5248.757903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5249.755891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5250.631818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5251.542773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5252.467134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5253.392074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5254.140627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5254.930534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5255.656335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5256.778278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5257.593739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5258.313211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5259.135576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5260.078721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5260.737524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5261.643406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5262.461713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5263.321228] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5263.329622] Lustre: Skipped 18 previous similar messages [ 5263.358443] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5263.367869] Lustre: Skipped 18 previous similar messages [ 5263.371686] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5263.383860] Lustre: Skipped 18 previous similar messages [ 5263.393423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5263.400864] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5263.408171] Lustre: Skipped 18 previous similar messages [ 5264.237571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5265.108248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5265.816297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5266.706688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5267.665886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5268.381460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5269.253722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5270.036197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5270.831657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5271.862316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5272.565951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5273.280543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5273.999941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5274.766197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5275.600753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5276.391592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5277.139249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5277.802192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5278.509775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5279.264194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5279.939901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5280.644092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5281.317067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5282.072696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5282.910699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5283.708455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5284.402973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5285.388682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5286.209216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5287.004433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5287.990700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5288.844300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5289.774884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5290.678362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5291.360912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5292.110090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5292.790702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5293.643133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5294.377587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5295.217418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5296.101551] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5296.105199] Lustre: Skipped 40 previous similar messages [ 5296.123176] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5296.136745] Lustre: Skipped 40 previous similar messages [ 5296.145172] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5296.156097] Lustre: Skipped 40 previous similar messages [ 5296.161110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5296.183153] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5296.189348] Lustre: Skipped 40 previous similar messages [ 5297.026372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5297.830166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5298.696604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5299.823324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5300.988765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5302.175641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5303.349235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5304.352162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5305.416461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5306.282911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5307.385312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5308.598383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5309.500843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5310.344643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5311.344307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5312.380677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5313.388545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5314.652867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5315.506109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5316.431857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5317.393935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5318.353514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5319.175470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5320.159700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5321.067318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5322.022749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5323.019550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5323.910815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5324.680331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5325.756714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5326.662743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5327.482661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5328.309193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5329.150257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5329.919274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5330.761454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5331.675510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5332.769484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5333.764926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5334.681227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5335.455793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5336.391071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5337.231348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5338.201111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5339.251571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5340.203733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5341.370957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5342.372865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5343.256964] LustreError: 172140:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5343.270394] LustreError: 172140:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5343.274177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5344.151220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5344.991661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5345.722061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5346.630418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5347.519712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5348.426054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5349.381252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5350.475871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5351.237116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5352.217877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5352.939169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5353.905675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5355.010063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5355.826593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5356.730539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5357.654434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5358.657950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5359.650308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5360.588890] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5360.602167] Lustre: Skipped 67 previous similar messages [ 5360.618373] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5360.631601] Lustre: Skipped 67 previous similar messages [ 5360.638788] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5360.643319] Lustre: Skipped 67 previous similar messages [ 5360.648419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5360.658600] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5360.665555] Lustre: Skipped 67 previous similar messages [ 5361.503515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5362.429967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5363.456389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5364.377495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5365.167411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5366.077980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5366.946828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5367.967580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5368.889120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5369.819287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5370.976188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5371.790735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5372.684815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5373.406777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5374.332733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5375.280994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5376.405367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5377.389505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5378.479238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5379.615485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5380.690562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5381.617886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5382.791347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5383.913864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5384.817178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5385.721770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5386.718887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5387.671661] LustreError: 172140:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5387.681369] LustreError: 172140:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5387.684377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5388.665820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5389.603068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5390.512475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5391.471668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5392.444907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5393.490823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5394.443689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5395.368971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5396.124747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5396.993809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.965200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5398.892918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5399.874664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5400.767837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5401.639494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5402.536219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5403.471284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5404.436377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5405.563564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5406.565676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5407.392298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.033954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.794392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5409.772606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5410.688864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5411.450745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5412.395674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5413.213480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5414.291572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5415.142976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5416.111339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5416.959155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5417.904869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.776661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5419.715872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5420.511247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5421.530793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5422.396658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5423.434046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5424.181389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5424.890941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5425.706306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5426.456259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5427.627692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5428.503141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5429.655460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5430.525610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5431.465955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5432.336682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5433.298255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5434.490828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5435.701426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5436.818618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5437.981922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5439.093097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5439.774804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5440.665544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5441.557069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5442.733388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5443.533870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5444.555230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5445.292516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5446.129203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5446.880366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5447.891758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5448.810174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5449.556106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5450.356702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5451.164353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5452.026442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5452.755501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5453.465770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5454.367164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5455.159778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5455.934720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5456.722882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5457.744314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5458.596656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5459.285911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5460.179967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.120242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.932079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5462.822495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5463.570250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5464.524393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5465.354634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5466.415388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.686624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5468.613549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.536746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5470.391046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5471.244777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5472.162318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.096714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5474.025604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.067345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.918521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5476.859030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5477.799455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5478.626389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5479.673581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5480.493971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.359222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.244254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5483.022818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.313219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.361908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5486.272256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5487.270605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.186346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.144982] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5489.152749] Lustre: Skipped 138 previous similar messages [ 5489.193215] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5489.215569] Lustre: Skipped 138 previous similar messages [ 5489.226283] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5489.235863] Lustre: Skipped 138 previous similar messages [ 5489.242989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.258618] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5489.267020] Lustre: Skipped 138 previous similar messages [ 5490.247313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.226700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.148049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.087353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.857176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5494.699080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.673702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5496.381363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.213566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5498.099445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.137728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.170587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.069949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.825687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5502.636112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5503.367200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.280314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.161202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.067447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.872965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.684574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.775760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5509.743453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.803540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5511.690393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.590542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.484449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5514.336220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.197171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.996931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5516.848650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.673663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.502994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5519.261272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.080030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.105404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.938767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5522.832344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.766603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.782739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5525.654684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5526.505699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.354917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.184786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.021746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.879595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5530.711371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.569220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.505198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5533.509592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.470850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5535.332688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.330877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5537.439201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.390562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5539.453949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.477618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5541.565140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.659060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.756467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.653807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.643771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.814886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.790541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.763116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.580410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.654557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.434074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.291295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.232609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.151795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.040561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.844309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5556.682211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.539276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.251701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.193704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.005918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.876620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.608835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.468868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.284167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.402814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5565.353194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.152427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.965876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.801255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.717205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.645867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.476891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.359326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.201317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.000469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.887646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.852394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.547441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.446423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.340517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.095252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.866968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.631074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.459168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.299251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.998593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.789210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.661518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.701379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.639108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5586.465645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.446793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.303225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.178040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.187110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.937526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.860281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.742305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.636057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.422699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.304962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.201689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.007897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.943913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.961922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.855151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.746966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.549326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.320850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.121706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.107245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.932067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.946345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.826907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.854643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.795313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.838901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.798485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.712265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.617957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.496278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.392422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.325779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.232246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.115233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.196840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.284674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.345797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.445794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.221247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.071603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.850598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.716246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.616144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.491628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.468908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.509231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.320145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.266433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.107162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.763390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.604453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.590308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.674538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.521078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.315307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.128703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.782924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.444261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.250706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.034074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.895806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.806601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.665218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.518677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.500768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.440596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.352909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.184261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.045447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.827253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.669510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.577170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.412710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.220155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.041275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.777701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.424201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.177252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.110612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.818901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.636967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.389633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.226795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.886815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.818558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.558474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.321708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.200733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.984301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.807337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.553598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.437504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.202698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.134442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.934474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.846644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.572808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.481845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.335711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.264528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.120182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.934641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.685497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.595104] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.539387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.288277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.989704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.912217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.638519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.492778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.319749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.120703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.925983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.735980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.498946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.242668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.968236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.767835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.671071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.411707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.213725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.121042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.935288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.710613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.498398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.275341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.100914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.957549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.635884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.400586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.066572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.707615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.542954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.283925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.060668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.797027] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.540087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.298847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.973799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.740151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.460550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.257771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.022791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.910553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.799400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.614688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.439951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.245293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.916403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5712.707153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.461175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.323979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.224478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.066722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.874204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.817769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.725858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.578096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.454353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.290633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.962686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.702078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.461124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.231627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.947963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.772880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.604577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.486944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.340732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.223308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.118869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.901789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.776609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.624941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.339485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.097649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.814072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.751568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.609621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.478881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.334286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.156866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.917501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5740.732523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.504602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.235615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.997201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.872541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.674578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.496970] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5745.505167] Lustre: Skipped 298 previous similar messages [ 5745.517344] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5745.535187] Lustre: Skipped 298 previous similar messages [ 5745.545156] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5745.556204] Lustre: Skipped 298 previous similar messages [ 5745.564572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.581189] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5745.594366] Lustre: Skipped 298 previous similar messages [ 5746.372218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.102685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.854308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.585742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.348248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5750.262870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.103492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.941083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.662639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.586670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.241621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.142799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.750917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.458318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.244765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.041113] LustreError: 172140:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5758.050572] LustreError: 172140:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5758.055937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.885138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.883337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.737253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.575322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.379544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.332317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.107324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.969618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.636549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.791640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.639107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.369091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.136487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.860123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.652364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.758077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.808093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.857762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.774681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.931178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.078438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.871194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.687168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.542177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.257893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.142237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.116433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.937350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.752428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.591113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.223632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.993469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.650834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.471929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.283600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.105886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.794562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.467563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.113976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.907812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.782048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.696243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.434715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.458616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.370443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.347724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.406743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.406524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.128537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.817748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.688595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.504750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.306348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.086261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.929792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.615917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.489488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.170244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.830880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.532776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.443154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.369873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.358853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.224087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.133310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.910509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5814.698504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.524610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.420402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.289617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.203684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.096551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.274658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.165793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.060989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.145073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.116765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.084151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.146496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.507318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.397581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.486528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.696259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.875853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.993486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.984538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5835.386220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.445627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.627476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.641362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.744509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.938203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.831870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.160305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5844.309654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.342699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.361521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.303114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.576170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.844817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.073774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.936140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.023615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.749965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.461434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.322923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.336491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5857.449412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.429323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.493167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.434192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.299131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.054739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.909655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.907786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.073793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.856272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.856373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.715505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.790345] LustreError: 172140:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5868.799818] LustreError: 172140:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5868.822762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.899540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.724308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.516989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.295025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.025987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.707988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.501462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5875.222141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5875.955779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.628393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.566726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.318322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.112124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.857469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.741130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5881.587727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5882.506200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.304569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.015563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.750576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.508614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.321141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.119598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.921200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.831833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.984646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.918629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.852029] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.671453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.639328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.356338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.219646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.034642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.970201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.902883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.894889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.729933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.642783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5901.460449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.228812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.928831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.783316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.494454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.360562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.176585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.113507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.954921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.830988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.557337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.564421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.284979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.164165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.956759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.720410] LustreError: 172140:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5913.728266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.729575] LustreError: 172140:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5914.521973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.335796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.297165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.114171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.867446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.597245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5919.350837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.073958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.879636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5921.633352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.464192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5923.318475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.189159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.988414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.809644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.409456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5927.036343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5927.861708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5928.717554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5929.540484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5930.302626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.072850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.831539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.525590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5933.351193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5934.242691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5935.009819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5935.814895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5936.497186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5937.412605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5938.308743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.215969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.124449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5941.006323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5941.842938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.757536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5943.615182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5944.577704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5945.543530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.464266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5947.355786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.194044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.094884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.981438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5950.801061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5951.759769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5952.782783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.782675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5954.799775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5955.616352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5956.373381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5957.506732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5959.020502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5960.394938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5961.991048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5963.339608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5964.897492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5966.409742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.816508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5969.358597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5970.821431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5972.695577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5974.616070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.524246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.176790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.543231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5980.953969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5982.204928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5983.551362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.717286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.059153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5987.322563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.190627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5989.613064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5990.380777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.616391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5992.478965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5993.376953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5994.242797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5995.438433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5996.347322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5997.295576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.319203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5999.540836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.751342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6002.033925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6003.610779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6004.953899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6005.611430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.451661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6007.288647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6008.486621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6009.526556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6010.926970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6012.158685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6013.376923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.631117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6015.956972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6016.757733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6017.581338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6018.554511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6019.607224] LustreError: 172140:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6019.626969] LustreError: 172140:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6019.636338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6020.725813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6021.747430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6022.875672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6023.854521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6024.891518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6025.628765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6026.701764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6027.673286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6028.946276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6030.230659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6031.284488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6032.387648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6033.396837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.684761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6035.835472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6036.728721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6037.821224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.690952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6039.710813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6040.494636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6041.605833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6042.891075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6044.133220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6045.746714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6046.911555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6047.925057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6049.109044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6050.148199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6050.832966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6051.630787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6052.393485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6053.294604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6054.033204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6054.998835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.042594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.948403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6057.998458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6059.138493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6060.250519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6061.531190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6062.699599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6063.915918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.884803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6066.119144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6067.061468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6068.156945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6069.290844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6070.401878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6071.681594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6072.862703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6074.211503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.423492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6076.443031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6077.656503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6078.750529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6079.998043] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6080.977491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6082.142420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6083.068218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6083.989059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6085.077510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6085.997629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6086.852657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6087.730193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6088.579219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6089.611687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6090.502747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6091.467320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6092.357592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6093.287037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6094.302487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6095.252917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6095.980452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6096.933162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6097.784614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6098.598672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6099.491318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6100.588991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6101.457542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6102.391345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6103.256831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6104.161982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6104.987907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6105.845430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6106.635958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6107.590459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6108.736207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6109.651848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6110.673209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6111.352932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6112.345898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6113.233323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6114.068482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6114.840525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6115.821305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6116.836682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6118.002514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6118.847921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6119.880828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6120.843289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6121.751562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6122.572819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6123.445588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6124.272709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6125.252738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6126.026600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6126.775202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6127.528401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6128.361524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6129.260583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6130.214588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6131.003242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6131.957486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6132.794345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6133.608555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6134.477537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6135.442684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6136.258253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6137.356025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6138.204670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6482.782927] LustreError: 174239:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6482.809261] LustreError: 172140:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 6905.833902] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6905.836241] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6905.842986] LustreError: Skipped 1 previous similar message [ 6905.858796] Lustre: Skipped 412 previous similar messages [ 6905.867084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6907.372719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6907.381651] Lustre: Skipped 1 previous similar message [ 6912.481644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6912.491507] Lustre: Skipped 1 previous similar message [ 6917.604276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6917.607028] Lustre: Skipped 1 previous similar message [ 6920.160136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6920.349356] Lustre: server umount lustre-OST0000 complete [ 6925.281194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6925.292391] LustreError: Skipped 5 previous similar messages [ 6925.304093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6930.633854] Lustre: server umount lustre-MDT0000 complete [ 6932.963375] LustreError: 172848:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6932.988290] LustreError: 172848:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6933.443319] LustreError: 211836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787586492 with bad export cookie 8641753349967437874 [ 6933.459673] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6933.461409] LustreError: 211836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6933.807367] Lustre: server umount lustre-MDT0001 complete [ 6944.082695] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [ 6946.518639] Key type lgssc unregistered [ 6946.843543] LNet: 227330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6946.857805] LNetError: 227330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6946.894767] LNet: Removed LNI 192.168.206.154@tcp [ 6947.673960] Key type .llcrypt unregistered [ 6947.676883] Key type ._llcrypt unregistered [ 6972.257880] Key type ._llcrypt registered [ 6972.262477] Key type .llcrypt registered [ 6972.468986] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 6974.283446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6974.423353] alg: No test for adler32 (adler32-zlib) [ 6975.771664] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 6976.100665] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [ 6977.800234] Key type lgssc registered [ 6978.771622] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6987.205706] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6987.233705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6988.768490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6992.737933] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 6999.011793] LustreError: 228745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6999.037326] LustreError: 228745:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7002.524810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7003.166100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7008.657949] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7017.090460] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7022.983263] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7028.744632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7029.021028] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7034.178455] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 7034.419167] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7040.992584] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7048.496580] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7049.944956] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7049.954435] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 7049.961197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7049.975955] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7049.976593] LustreError: 228765:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 7052.774897] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7052.775086] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7052.779274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7052.798903] Lustre: Skipped 2 previous similar messages [ 7053.486354] Lustre: server umount lustre-OST0000 complete [ 7070.016187] LustreError: 228765:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 7085.954151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7086.213034] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7091.175506] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7092.334231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7092.343172] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7092.343627] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7092.356634] Lustre: Skipped 1 previous similar message [ 7097.411444] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7098.749593] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7103.888196] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8b87d0425800.ost_server_uuid 50 [ 7105.570159] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8b87d0425800.ost_server_uuid in FULL state after 0 sec [ 7111.137463] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7111.149357] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7111.167579] Lustre: Skipped 1 previous similar message [ 7111.188755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7113.189485] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7113.192673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7113.198487] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7113.223505] Lustre: Skipped 1 previous similar message [ 7115.474356] Lustre: server umount lustre-OST0000 complete [ 7120.867449] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7120.880370] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7120.903440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7128.545537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7128.555683] Lustre: Skipped 3 previous similar messages [ 7133.152189] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7133.334240] Lustre: server umount lustre-MDT0000 complete [ 7133.668583] LustreError: 231592:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7133.686714] LustreError: 231592:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7136.515908] LustreError: 228727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787586695 with bad export cookie 15668187654960532438 [ 7136.521264] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7136.524685] LustreError: 228727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7136.822293] Lustre: server umount lustre-MDT0001 complete [ 7145.151344] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [ 7147.786887] Key type lgssc unregistered [ 7148.044187] LNet: 233957:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7148.053255] LNetError: 233957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7148.086880] LNet: Removed LNI 192.168.206.154@tcp [ 7148.872655] Key type .llcrypt unregistered [ 7148.874711] Key type ._llcrypt unregistered [ 7182.271437] Key type ._llcrypt registered [ 7182.273752] Key type .llcrypt registered [ 7182.366593] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 7183.576768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7183.635357] alg: No test for adler32 (adler32-zlib) [ 7184.846485] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 7185.169479] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [ 7186.872497] Key type lgssc registered [ 7188.100847] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7197.106980] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7197.127841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7198.509634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7202.382757] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7208.932886] LustreError: 235688:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7208.949442] LustreError: 235688:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7209.882415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7210.133948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7213.766710] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7215.960858] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7221.190369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7221.570618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7223.612756] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 7226.897640] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7234.459145] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7234.626564] Lustre: lustre-OST0001: new disk, initializing [ 7234.630538] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7234.636980] Lustre: 237993:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7234.711291] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7240.241327] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7242.359259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7242.367845] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7242.519355] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7250.648845] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7257.440593] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7269.584722] Lustre: server umount lustre-MDT0000 complete [ 7271.913815] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7271.922291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7271.933579] LustreError: 235689:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7272.853714] LustreError: 236423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787586831 with bad export cookie 6632047655055306718 [ 7272.868624] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7272.871289] LustreError: 236423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7273.442307] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7273.450691] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7273.453032] LustreError: 235685:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7273.454426] Lustre: Skipped 3 previous similar messages [ 7273.477372] Lustre: Skipped 1 previous similar message [ 7275.208751] Lustre: server umount lustre-MDT0001 complete [ 7278.641069] Lustre: server umount lustre-OST0000 complete [ 7282.121330] Lustre: server umount lustre-OST0001 complete [ 7302.247864] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 7312.149443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7312.551581] LustreError: 241099:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7312.568262] LustreError: 241099:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7312.642671] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7317.819272] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7317.987406] LustreError: 241100:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7325.732340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7326.082268] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7330.356458] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7333.457546] Lustre: 242242:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7339.100750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7345.792429] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7350.843311] LustreError: 242595:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7350.858684] LustreError: 242595:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7355.110644] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7355.376662] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7355.391513] Lustre: Skipped 1 previous similar message [ 7357.443813] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49730 to 0x280000401:49760) [ 7357.447580] Lustre: mylustre-OST0001: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 7362.212800] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7369.691471] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7372.843796] Lustre: 244111:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7382.503070] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7382.509282] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7382.519546] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7388.130456] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7388.139252] Lustre: Skipped 4 previous similar messages [ 7388.568937] Lustre: server umount mylustre-MDT0000 complete [ 7391.714151] LustreError: 241074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787586950 with bad export cookie 6632047655055309525 [ 7391.719268] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7391.738560] LustreError: 241074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7392.017307] Lustre: server umount mylustre-MDT0001 complete [ 7405.876767] Lustre: server umount mylustre-OST0000 complete [ 7419.738366] Lustre: server umount mylustre-OST0001 complete [ 7440.414533] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 7450.075311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7450.692385] LustreError: 246420:0:(ldlm_lib.c:1190:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7450.788097] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7454.703386] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7463.384972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7468.099136] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7470.572029] Lustre: 247561:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7477.689596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7483.730492] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7489.263904] LustreError: 247917:0:(ldlm_lib.c:1190:target_handle_connect()) tfs-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7489.294624] LustreError: 247917:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7491.608083] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7496.006179] Lustre: tfs-OST0001: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [ 7496.035448] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49762 to 0x280000401:49792) [ 7497.770672] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7504.970268] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7508.582152] Lustre: 249432:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7518.962941] Lustre: server umount tfs-MDT0000 complete [ 7520.226348] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7520.234164] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7520.243224] Lustre: Skipped 3 previous similar messages [ 7522.490846] LustreError: 247564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787587081 with bad export cookie 6632047655055311415 [ 7522.504682] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7522.509820] LustreError: 247564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7522.930891] Lustre: server umount tfs-MDT0001 complete [ 7537.256526] Lustre: server umount tfs-OST0000 complete [ 7551.252236] Lustre: server umount tfs-OST0001 complete [ 7569.884851] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 7580.198600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7580.704199] LustreError: 251683:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7580.726425] LustreError: 251683:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7580.843673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7580.856991] Lustre: Skipped 3 previous similar messages [ 7585.499900] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7593.785530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7598.309613] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7600.982170] Lustre: 252825:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7606.202435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7611.857202] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7620.256708] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7625.671209] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49762 to 0x280000401:49824) [ 7625.692377] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [ 7627.032749] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7635.096961] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7638.960911] Lustre: 254694:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7646.349305] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 12:00:03 (1787587203) [ 7665.634243] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7665.639830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7665.651163] Lustre: Skipped 3 previous similar messages [ 7665.658613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7669.316890] Lustre: server umount lustre-MDT0000 complete [ 7672.839766] LustreError: 251661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787587231 with bad export cookie 6632047655055313081 [ 7672.846506] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7672.852560] LustreError: 251661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7676.902226] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7676.925621] Lustre: Skipped 3 previous similar messages [ 7676.933710] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7676.942945] Lustre: Skipped 5 previous similar messages [ 7687.136594] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7687.422189] Lustre: server umount lustre-MDT0001 complete [ 7697.651439] Lustre: server umount lustre-OST0000 complete [ 7709.198897] Lustre: server umount lustre-OST0001 complete [ 7716.716706] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [ 7728.549730] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 7763.238039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7763.419601] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7763.447714] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7763.507752] Lustre: lustre-MDT0000: new disk, initializing [ 7763.576634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 188-180 [ 7763.582751] Lustre: Skipped 3 previous similar messages [ 7763.594790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7767.861441] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7777.137688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7777.249604] Lustre: 257887:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7777.289335] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7777.292691] Lustre: Skipped 1 previous similar message [ 7777.353466] Lustre: lustre-MDT0001: new disk, initializing [ 7777.458556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7777.471891] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7781.936656] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7791.730296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7792.015602] Lustre: lustre-OST0000: new disk, initializing [ 7792.026346] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7792.033752] Lustre: 259471:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7793.249481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7793.257749] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7793.416996] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7797.626502] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7807.532473] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7807.707197] Lustre: lustre-OST0001: new disk, initializing [ 7807.710111] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7807.716071] Lustre: 260341:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7809.566377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7809.578904] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7809.693247] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7813.051415] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 7823.821592] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 12:03:01 (1787587381) [ 7836.745364] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 12:03:14 (1787587394) [ 7855.075286] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7855.077382] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7855.079062] LustreError: Skipped 1 previous similar message [ 7855.080079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7855.090325] Lustre: Skipped 2 previous similar messages [ 7855.102785] Lustre: Skipped 5 previous similar messages [ 7857.353105] Lustre: server umount lustre-OST0000 complete [ 7860.719596] LustreError: 259562:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7860.747213] LustreError: 259562:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 7864.804860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7867.470303] Lustre: server umount lustre-MDT0000 complete [ 7870.824956] LustreError: 257882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787587429 with bad export cookie 6632047655055314635 [ 7870.832083] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7870.836091] LustreError: 257882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7876.065816] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7876.071155] Lustre: Skipped 5 previous similar messages [ 7885.280113] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7885.560410] Lustre: server umount lustre-MDT0001 complete [ 7896.757949] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [ 7975.812569] Lustre: server umount lustre-OST0001 complete [ 7982.851315] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [ 7999.226991] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 8040.269936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8040.448273] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8040.494350] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8040.571953] Lustre: lustre-MDT0000: new disk, initializing [ 8040.638818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8040.645747] Lustre: Skipped 3 previous similar messages [ 8040.664408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8046.405638] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 8057.909726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8058.034576] Lustre: 265204:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8058.088533] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8058.093162] Lustre: Skipped 1 previous similar message [ 8058.246186] Lustre: lustre-MDT0001: new disk, initializing [ 8058.423162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8058.445749] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8063.702313] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 8077.682710] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8085.799175] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8095.125782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8095.608600] Lustre: lustre-OST0000: new disk, initializing [ 8095.616637] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8095.627779] Lustre: 267174:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8096.800985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8096.819542] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8096.967079] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8103.225994] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 8114.114886] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8122.857878] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8122.863707] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8122.867073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8122.867082] Lustre: Skipped 2 previous similar messages [ 8122.891117] Lustre: Skipped 5 previous similar messages [ 8122.920261] LustreError: Skipped 1 previous similar message [ 8128.241675] Lustre: server umount lustre-OST0000 complete [ 8140.645424] Lustre: server umount lustre-MDT0000 complete [ 8143.333542] LustreError: 265214:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8143.367511] LustreError: 265214:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 8144.020919] LustreError: 265997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787587702 with bad export cookie 6632047655055316105 [ 8144.028067] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8144.029834] LustreError: 265997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8144.670956] Lustre: server umount lustre-MDT0001 complete [ 8155.915704] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [ 8158.539601] Key type lgssc unregistered [ 8158.847773] LNet: 268938:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8158.862281] LNetError: 268938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8158.888810] LNet: Removed LNI 192.168.206.154@tcp [ 8159.560331] Key type .llcrypt unregistered [ 8159.563229] Key type ._llcrypt unregistered [ 8184.996175] Key type ._llcrypt registered [ 8184.999280] Key type .llcrypt registered [ 8185.106665] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 8186.049815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8186.182089] alg: No test for adler32 (adler32-zlib) [ 8187.259589] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 8187.491990] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [ 8189.160169] Key type lgssc registered [ 8190.274751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8197.970110] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8197.988698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8199.424441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8203.685368] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 8209.892210] LustreError: 270402:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8209.911251] LustreError: 270402:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8212.287816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8212.625028] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8217.281949] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 8224.983191] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8231.059853] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8237.757924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8238.148349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8244.350476] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 8244.870918] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 8252.258971] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8253.516444] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 8257.447277] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8257.454770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8257.478056] Lustre: Skipped 1 previous similar message [ 8262.637469] LustreError: 272224:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8262.655015] LustreError: 272224:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8262.714896] Lustre: server umount lustre-OST0000 complete [ 8267.747447] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8267.759364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8267.764149] Lustre: Skipped 1 previous similar message [ 8272.870918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8272.879065] Lustre: Skipped 1 previous similar message [ 8273.009862] Lustre: server umount lustre-MDT0000 complete [ 8276.507356] LustreError: 270382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787587835 with bad export cookie 16930981812884876699 [ 8276.511184] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8276.521349] LustreError: 270382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8276.805288] Lustre: server umount lustre-MDT0001 complete [ 8287.414561] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [ 8290.161691] Key type lgssc unregistered [ 8290.519405] LNet: 273993:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8290.525879] LNetError: 273993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8290.539611] LNet: Removed LNI 192.168.206.154@tcp [ 8291.377679] Key type .llcrypt unregistered [ 8291.382778] Key type ._llcrypt unregistered [ 8306.573214] Key type ._llcrypt registered [ 8306.577466] Key type .llcrypt registered [ 8306.737599] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 8307.788664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8307.846774] alg: No test for adler32 (adler32-zlib) [ 8309.072175] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 8309.378570] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [ 8311.113537] Key type lgssc registered [ 8312.270645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8321.502329] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8321.533874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8337.184597] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8342.304483] LustreError: 275187:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.154@tcp: failed processing log, type 4: rc = -110 [ 8368.992657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8375.148189] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 8382.654643] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8384.752683] Lustre: server umount lustre-OST0000 complete [ 8400.204490] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [ 8402.859807] Key type lgssc unregistered [ 8403.141983] LNet: 276820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8403.151759] LNetError: 276820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8403.176426] LNet: Removed LNI 192.168.206.154@tcp [ 8403.962573] Key type .llcrypt unregistered [ 8403.964958] Key type ._llcrypt unregistered [ 8446.628118] Key type ._llcrypt registered [ 8446.631767] Key type .llcrypt registered [ 8446.826569] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 8448.545039] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8448.645302] alg: No test for adler32 (adler32-zlib) [ 8449.710935] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 8450.075347] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [ 8451.809160] Key type lgssc registered [ 8453.205851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8466.803919] loop0: detected capacity change from 0 to 204800000 [ 8467.049117] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8467.067145] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8467.507475] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8468.519192] loop0: detected capacity change from 0 to 204800000 [ 8468.804306] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8468.820169] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8469.168301] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8470.090664] loop0: detected capacity change from 0 to 204800000 [ 8470.284909] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8470.306335] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8470.409711] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8471.602614] loop0: detected capacity change from 0 to 204800000 [ 8471.852929] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8473.140944] loop0: detected capacity change from 0 to 204800000 [ 8473.180826] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8478.455580] loop0: detected capacity change from 0 to 204800000 [ 8478.487602] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8482.724629] loop0: detected capacity change from 0 to 204800000 [ 8482.757746] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8486.798904] loop0: detected capacity change from 0 to 204800000 [ 8486.831738] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8490.457342] loop0: detected capacity change from 0 to 204800000 [ 8490.512350] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8490.569955] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8490.657949] Lustre: 280237:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8490.666305] Lustre: 280237:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 8490.675509] Lustre: 280237:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8490.685617] Lustre: 280237:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8490.699679] Lustre: 280237:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 8490.714484] Lustre: 280237:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 8490.728362] Lustre: 280237:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8490.742322] Lustre: 280237:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 8490.758379] Lustre: 280237:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 8490.775464] Lustre: 280237:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 8490.799564] Lustre: 280237:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 8490.812665] Lustre: 280237:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 8490.830570] Lustre: 280237:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 8490.843966] Lustre: 280237:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 8496.743309] Lustre: server umount MGS complete [ 8497.741063] loop0: detected capacity change from 0 to 204800000 [ 8497.899093] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8498.478916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8498.564867] Lustre: lustre-MDT0000: Aborting client recovery [ 8499.703654] loop1: detected capacity change from 0 to 204800000 [ 8499.923719] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8499.987320] Lustre: 280668:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8499.995810] Lustre: 280668:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 8500.004242] Lustre: 280668:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8500.013387] Lustre: 280668:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 8500.021155] Lustre: 280668:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8500.030267] Lustre: 280668:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8500.037834] Lustre: 280668:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 8500.059621] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8500.232619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8500.280089] Lustre: lustre-MDT0001: Aborting client recovery [ 8501.118362] loop2: detected capacity change from 0 to 204800000 [ 8501.269364] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8501.372894] Lustre: 280748:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8501.395530] Lustre: 280748:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8501.408532] Lustre: 280748:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8501.430643] Lustre: 280748:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8501.450771] Lustre: 280748:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8501.457953] Lustre: 280748:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8501.464943] Lustre: 280748:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8501.562430] Lustre: Found index 0 for lustre-OST0000, updating log [ 8501.627668] Lustre: 280767:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8501.708669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8501.742044] Lustre: lustre-OST0000: Aborting recovery [ 8502.703039] loop3: detected capacity change from 0 to 204800000 [ 8502.895878] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8502.968833] Lustre: 280832:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8502.975916] Lustre: 280832:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8502.984081] Lustre: 280832:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8502.992876] Lustre: 280832:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8503.001485] Lustre: 280832:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8503.010745] Lustre: 280832:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8503.018794] Lustre: 280832:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8503.035283] Lustre: Found index 1 for lustre-OST0001, updating log [ 8503.076630] Lustre: 280838:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8503.143754] Lustre: lustre-OST0001: Aborting recovery [ 8508.278963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8508.315295] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8508.325530] LustreError: 280612:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 8508.355630] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 8508.441623] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 8510.320273] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 8510.322370] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 8511.543514] LustreError: 280579:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 1077us: rc = -2 [ 8512.078168] LustreError: 280579:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0001: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0001 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0001/identity_upcall, time 1305us: rc = -2 [ 8512.108861] LustreError: 280579:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 14 previous similar messages [ 8525.793655] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8525.801174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8525.818498] Lustre: Skipped 2 previous similar messages [ 8525.846955] Lustre: Skipped 3 previous similar messages [ 8530.919755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8530.924997] Lustre: Skipped 3 previous similar messages [ 8531.389913] Lustre: server umount lustre-MDT0000 complete [ 8532.474323] LustreError: 280564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787588091 with bad export cookie 15716417243447933367 [ 8532.482061] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8532.494623] LustreError: 280564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8536.033595] LustreError: 280576:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8536.034190] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8536.035108] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8536.042341] LustreError: 280576:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8536.088260] Lustre: Skipped 2 previous similar messages [ 8537.057931] LustreError: 280577:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8537.081829] LustreError: 280577:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8538.962259] Lustre: server umount lustre-MDT0001 complete [ 8546.217501] Lustre: server umount lustre-OST0000 complete [ 8553.478784] Lustre: server umount lustre-OST0001 complete [ 8562.098140] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 12:15:19 (1787588119) [ 8589.867931] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [ 8599.845916] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 8633.694663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8633.998940] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8634.041577] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8634.146555] Lustre: lustre-MDT0000: new disk, initializing [ 8634.194124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8634.197879] Lustre: Skipped 1 previous similar message [ 8634.211623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8638.758781] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 8648.527246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8648.656688] Lustre: 284909:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8648.746702] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8648.750400] Lustre: Skipped 1 previous similar message [ 8648.843424] Lustre: lustre-MDT0001: new disk, initializing [ 8648.902925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8648.934029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8648.944604] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8653.657796] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 8661.615822] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8666.636589] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8672.888982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8673.059442] Lustre: lustre-OST0000: new disk, initializing [ 8673.063768] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8673.070167] Lustre: 286877:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8673.191514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8678.497078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8678.518763] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8678.612920] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8679.038750] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 8687.345610] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8692.339309] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 8703.027449] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 8710.178671] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 8719.103719] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 8760.811951] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8760.814628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8760.816355] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8760.838171] Lustre: Skipped 1 previous similar message [ 8760.867467] Lustre: Skipped 4 previous similar messages [ 8765.925741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8766.352872] Lustre: server umount lustre-OST0000 complete [ 8771.045240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8771.065250] Lustre: Skipped 1 previous similar message [ 8776.161723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8776.171417] Lustre: Skipped 4 previous similar messages [ 8776.439369] Lustre: server umount lustre-MDT0000 complete [ 8780.021989] LustreError: 284901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787588338 with bad export cookie 15716417243447936090 [ 8780.023613] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8780.028842] LustreError: 284901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8780.517979] Lustre: server umount lustre-MDT0001 complete [ 8788.733821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8793.067970] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 8797.978813] Lustre: server umount MGS complete [ 8808.500151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8809.002843] LustreError: 290798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8809.043661] LustreError: 290811:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 8809.115071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8813.366145] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 8814.572637] LustreError: 290799:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8818.664725] LustreError: 290798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8821.903845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8826.259462] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 8833.205267] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8838.217625] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8844.316954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8844.686967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8844.693427] Lustre: Skipped 1 previous similar message [ 8850.445677] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 8857.882269] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8888.804730] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8888.814624] LustreError: Skipped 1 previous similar message [ 8888.823769] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8888.842638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8889.827362] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8893.710279] Lustre: server umount lustre-OST0000 complete [ 8899.043111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8899.054966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8899.073619] Lustre: Skipped 1 previous similar message [ 8912.352585] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8912.702825] Lustre: server umount lustre-MDT0000 complete [ 8914.407401] LustreError: 290793:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8914.432488] LustreError: 290793:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8916.685964] LustreError: 290779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787588475 with bad export cookie 15716417243447938036 [ 8916.688320] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8916.694567] LustreError: 290779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8916.993801] Lustre: server umount lustre-MDT0001 complete [ 8930.749583] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [ 8934.109682] Key type lgssc unregistered [ 8934.492383] LNet: 294747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8934.498166] LNetError: 294747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8934.511789] LNet: Removed LNI 192.168.206.154@tcp [ 8935.265974] Key type .llcrypt unregistered [ 8935.268109] Key type ._llcrypt unregistered [ 8973.295858] Key type ._llcrypt registered [ 8973.297993] Key type .llcrypt registered [ 8973.446098] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [ 8988.388701] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 8989.225792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8989.301584] alg: No test for adler32 (adler32-zlib) [ 8990.323837] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 8990.521411] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [ 8992.176201] Key type lgssc registered [ 8993.639506] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9025.705803] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9025.755436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9026.992927] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9027.033122] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9027.117273] Lustre: lustre-MDT0000: new disk, initializing [ 9027.206965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9027.221884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9031.313653] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 9040.885938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9041.041242] Lustre: 298705:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9041.103427] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9041.108955] Lustre: Skipped 1 previous similar message [ 9041.189362] Lustre: lustre-MDT0001: new disk, initializing [ 9041.286910] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9041.333223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9041.351126] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9045.820837] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 9055.529974] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9059.997574] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9066.552523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9066.775923] Lustre: lustre-OST0000: new disk, initializing [ 9066.783178] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9066.799525] Lustre: 300676:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9066.920710] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9072.917728] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 9074.802285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9074.818227] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9074.939064] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9082.094205] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9089.346726] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9100.889068] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9108.924751] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9162.211819] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9162.214189] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9162.224123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9162.239269] Lustre: Skipped 1 previous similar message [ 9163.712676] Lustre: server umount lustre-OST0000 complete [ 9167.329896] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9167.345767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9167.355526] Lustre: Skipped 2 previous similar messages [ 9172.451384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9172.462523] Lustre: Skipped 1 previous similar message [ 9173.475973] Lustre: server umount lustre-MDT0000 complete [ 9176.891996] LustreError: 298700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787588735 with bad export cookie 6382353754835702769 [ 9176.895952] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9176.908852] LustreError: 298700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9177.199786] Lustre: server umount lustre-MDT0001 complete [ 9186.187115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9190.748157] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 9195.714578] Lustre: server umount MGS complete [ 9205.959662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9206.405311] LustreError: 304602:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9206.493906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9210.660858] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 9211.878606] LustreError: 304590:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9211.889432] LustreError: 304590:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9216.997214] LustreError: 304589:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9218.891236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9219.170919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9225.072119] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 9232.266470] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9238.373467] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9245.547746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9245.913211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9252.444600] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 9259.912507] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9293.282422] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9293.290971] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9293.303325] Lustre: Skipped 1 previous similar message [ 9293.315124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9294.311472] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9296.533992] Lustre: server umount lustre-OST0000 complete [ 9300.962174] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9300.973157] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9300.982556] Lustre: Skipped 1 previous similar message [ 9300.987412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9300.994203] Lustre: Skipped 2 previous similar messages [ 9309.667710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9309.680601] Lustre: Skipped 3 previous similar messages [ 9314.272190] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9314.421229] Lustre: server umount lustre-MDT0000 complete [ 9314.786652] LustreError: 305318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9314.799370] LustreError: 305318:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9317.822663] LustreError: 304570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787588876 with bad export cookie 6382353754835704617 [ 9317.824495] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9317.836252] LustreError: 304570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9318.170560] Lustre: server umount lustre-MDT0001 complete [ 9329.947926] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [ 9333.068214] Key type lgssc unregistered [ 9333.376829] LNet: 308539:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9333.390964] LNetError: 308539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9334.446594] LNet: Removed LNI 192.168.206.154@tcp [ 9335.384062] Key type .llcrypt unregistered [ 9335.388320] Key type ._llcrypt unregistered [ 9355.824189] Key type ._llcrypt registered [ 9355.826564] Key type .llcrypt registered [ 9355.934583] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [ 9370.157979] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 9371.506049] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9371.576089] alg: No test for adler32 (adler32-zlib) [ 9372.566706] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 9372.847610] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [ 9374.528354] Key type lgssc registered [ 9375.550717] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9400.625606] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [ 9415.265747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9415.312063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9416.531397] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9416.567791] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9416.662527] Lustre: lustre-MDT0000: new disk, initializing [ 9416.755047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9416.799340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9421.490975] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 9432.989674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9433.307925] Lustre: lustre-OST0000: new disk, initializing [ 9433.311827] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9433.316494] Lustre: Skipped 1 previous similar message [ 9433.324154] Lustre: 312594:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9433.422879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9440.232511] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [ 9441.956143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9441.972265] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9442.002904] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10256.870166] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10256.893304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10260.306919] Lustre: server umount lustre-MDT0000 complete [10263.906519] LustreError: 312593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787589822 with bad export cookie 3589371568294733513 [10263.910976] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10263.913554] LustreError: 312593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10264.150633] Lustre: server umount lustre-OST0000 complete [10292.655931] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [10303.515357] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [10343.956211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10344.131532] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10344.198649] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10344.361568] Lustre: lustre-MDT0000: new disk, initializing [10344.531095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10344.555073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10349.477166] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [10359.702106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10359.862670] Lustre: 316927:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10359.927700] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10359.931233] Lustre: Skipped 1 previous similar message [10360.062929] Lustre: lustre-MDT0001: new disk, initializing [10360.163610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10360.216351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10360.225284] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10364.573807] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [10373.650741] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10378.814790] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10385.386978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10385.769045] Lustre: lustre-OST0000: new disk, initializing [10385.775934] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10385.787175] Lustre: 318892:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10385.902937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10392.149808] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [10392.673048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10392.691823] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10392.845482] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10401.308478] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10406.206500] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10406.357994] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10410.672643] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10410.889471] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10413.548195] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10413.551086] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10413.558669] LustreError: Skipped 1 previous similar message [10413.562825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10413.584675] Lustre: Skipped 1 previous similar message [10413.617382] Lustre: Skipped 1 previous similar message [10418.660518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10418.668199] Lustre: Skipped 1 previous similar message [10423.780676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10426.848148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10427.007690] Lustre: server umount lustre-OST0000 complete [10431.970749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10431.983103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10431.993500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10432.002412] Lustre: Skipped 1 previous similar message [10434.036314] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10436.950625] Lustre: server umount lustre-MDT0000 complete [10440.352683] LustreError: 318903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787589999 with bad export cookie 3589371568297674332 [10440.355976] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10440.363520] LustreError: 318903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10440.729561] Lustre: server umount lustre-MDT0001 complete [10445.102317] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [10447.137146] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 12:46:44 (1787590004) [10454.696369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10455.419512] LustreError: 321206:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10455.444715] LustreError: 321206:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10455.537412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10460.646104] LustreError: 321207:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10460.666445] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [10465.762905] LustreError: 321206:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10469.287738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10469.629874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10474.584749] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [10481.926186] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10487.525855] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10494.063806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10494.455387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10500.622600] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [10507.536965] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10513.421848] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10513.630769] Lustre: lustre-OST0001: new disk, initializing [10513.636944] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10513.641863] Lustre: 324042:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10513.717736] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10519.094385] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [10520.179727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10520.193493] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10520.246823] LustreError: 324463:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10521.317392] LustreError: 324463:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10521.327906] LustreError: 324463:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [10522.340822] LustreError: 324463:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10522.362232] LustreError: 324463:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [10524.394555] LustreError: 324471:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10524.414067] LustreError: 324471:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [10528.439498] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10528.617233] LustreError: 324471:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10528.628896] LustreError: 324471:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [10528.640211] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10533.687283] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8b87deb81000.ost_server_uuid 50 [10535.161516] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8b87deb81000.ost_server_uuid in FULL state after 0 sec [10536.942503] LustreError: 324463:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10536.955818] LustreError: 324463:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 15 previous similar messages [10540.111854] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b87deb81000.ost_server_uuid 50 [10541.384791] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8b87deb81000.ost_server_uuid in FULL state after 0 sec [10543.159258] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10556.395153] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10556.401696] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10556.416842] Lustre: Skipped 1 previous similar message [10556.434269] Lustre: Skipped 3 previous similar messages [10562.586978] Lustre: server umount lustre-OST0001 complete [10565.102031] LustreError: 323026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10571.233680] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10571.239987] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10571.749812] LustreError: 323026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10571.771965] LustreError: 323026:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10574.604493] Lustre: server umount lustre-OST0000 complete [10581.988110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10581.994493] Lustre: Skipped 5 previous similar messages [10584.768252] Lustre: server umount lustre-MDT0000 complete [10587.105451] LustreError: 324617:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10587.126907] LustreError: 324617:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10588.224269] LustreError: 321187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787590147 with bad export cookie 3589371568297675214 [10588.226880] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10588.242451] LustreError: 321187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10588.680582] Lustre: server umount lustre-MDT0001 complete [10598.703990] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [10601.612212] Key type lgssc unregistered [10601.929848] LNet: 326732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10601.941780] LNetError: 326732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10601.970583] LNet: Removed LNI 192.168.206.154@tcp [10602.859569] Key type .llcrypt unregistered [10602.866170] Key type ._llcrypt unregistered [10626.269853] Key type ._llcrypt registered [10626.273969] Key type .llcrypt registered [10626.382823] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [10627.250954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10627.270320] alg: No test for adler32 (adler32-zlib) [10628.409737] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [10628.695066] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [10630.370679] Key type lgssc registered [10631.570336] Lustre: Echo OBD driver; http://www.lustre.org/ [10639.267460] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10639.296442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10640.923501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10645.443313] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [10651.111473] LustreError: 328156:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10651.133979] LustreError: 328156:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10652.749417] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10656.239696] LustreError: 328157:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10661.014766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10661.359449] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10661.524865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10666.095807] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [10675.229564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10675.634605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10681.819866] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [10685.881048] LustreError: 329836:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10685.911411] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [10689.923724] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10692.071300] LustreError: 329835:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10692.098537] LustreError: 329835:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10697.194839] LustreError: 329834:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10697.227664] LustreError: 329834:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10697.251485] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10697.581339] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10702.862394] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [10704.623854] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [10712.114736] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10719.872623] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [10720.125392] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10825.698614] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10825.699496] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10825.708930] LustreError: Skipped 1 previous similar message [10825.711314] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10825.719651] Lustre: Skipped 1 previous similar message [10825.748673] Lustre: Skipped 1 previous similar message [10828.439051] Lustre: server umount lustre-OST0001 complete [10830.829842] LustreError: 330365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10830.852122] LustreError: 330365:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10834.401566] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10834.414684] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10834.443813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10835.429245] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10835.438135] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10840.478977] Lustre: server umount lustre-OST0000 complete [10845.665392] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10845.676486] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10845.695017] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10845.697506] Lustre: Skipped 2 previous similar messages [10850.602077] Lustre: server umount lustre-MDT0000 complete [10851.300657] LustreError: 328156:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10851.314839] LustreError: 328156:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10854.760596] LustreError: 328136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787590413 with bad export cookie 9412916999672921743 [10854.763718] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10854.773146] LustreError: 328136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10855.090764] Lustre: server umount lustre-MDT0001 complete [10866.697547] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [10869.978315] Key type lgssc unregistered [10870.392089] LNet: 333103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10870.413115] LNetError: 333103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10870.438350] LNet: Removed LNI 192.168.206.154@tcp [10871.312482] Key type .llcrypt unregistered [10871.315577] Key type ._llcrypt unregistered [10914.711151] Key type ._llcrypt registered [10914.714330] Key type .llcrypt registered [10914.847710] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [10915.779331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10915.864193] alg: No test for adler32 (adler32-zlib) [10916.983952] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [10917.195178] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [10918.832167] Key type lgssc registered [10919.934925] Lustre: Echo OBD driver; http://www.lustre.org/ [10930.319897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10930.344951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10931.996770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10937.278384] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [10941.442326] LustreError: 335395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10941.469444] LustreError: 335395:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10946.541858] LustreError: 335394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10946.915377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10947.403058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10952.307621] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [10955.206070] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10962.559906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10963.029216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10969.105153] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [10972.282742] LustreError: 336884:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10976.368861] LustreError: 337399:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10976.402068] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [10976.406846] LustreError: 337399:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10977.899870] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10978.138186] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10983.433077] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [10984.511432] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [10993.525041] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10999.253638] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11014.134106] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11014.139509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11014.160420] Lustre: Skipped 3 previous similar messages [11014.175938] Lustre: Skipped 3 previous similar messages [11016.170712] Lustre: server umount lustre-MDT0000 complete [11019.243194] LustreError: 335394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11019.269886] LustreError: 335394:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11019.954250] LustreError: 335374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787590578 with bad export cookie 12714713649898016014 [11019.964625] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11019.974492] LustreError: 335374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11020.269837] Lustre: server umount lustre-MDT0001 complete [11034.824852] Lustre: server umount lustre-OST0000 complete [11048.950698] Lustre: server umount lustre-OST0001 complete [11071.106069] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [11081.755279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11082.234296] LustreError: 340660:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11082.335161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11086.500721] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [11094.378313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11094.741534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11098.995349] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [11101.416121] Lustre: 341801:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11107.449674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11112.650933] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [11119.972516] LustreError: 342155:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11119.996960] LustreError: 342155:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11120.318558] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11120.611597] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11120.623181] Lustre: Skipped 1 previous similar message [11126.413857] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [11127.809991] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [11127.834451] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [11133.684876] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11136.938220] Lustre: 343671:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11143.146796] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11143.160103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11143.179548] Lustre: Skipped 2 previous similar messages [11143.187413] Lustre: Skipped 1 previous similar message [11147.764760] Lustre: server umount lustre-MDT0000 complete [11151.784552] LustreError: 340642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787590710 with bad export cookie 12714713649898017715 [11151.786693] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11151.793306] LustreError: 340642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11152.280748] Lustre: server umount lustre-MDT0001 complete [11168.045409] Lustre: server umount lustre-OST0000 complete [11169.636283] Lustre: 334658:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787590712/real 1787590712] req@ffff8c5b66e54380 x1874424471072512/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787590728 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11169.658943] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11169.668965] Lustre: Skipped 1 previous similar message [11172.009291] Lustre: server umount lustre-OST0001 complete [11175.478539] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [11177.028804] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [11178.825123] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 12:58:56 (1787590736) [11204.002631] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [11212.862893] loop0: detected capacity change from 0 to 1099511627776 [11213.494365] SGI XFS with ACLs, security attributes, quota, no debug enabled [11213.499553] *** VALIDATE xfs *** [11213.507689] XFS (loop0): Mounting V5 Filesystem [11213.529397] XFS (loop0): Ending clean mount [11214.905262] loop1: detected capacity change from 0 to 18691697672192 [11332.066504] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [11341.680568] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [11378.721566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11379.060894] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11379.117781] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11379.220881] Lustre: lustre-MDT0000: new disk, initializing [11379.300956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11379.321716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11384.189316] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [11394.216043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11394.335694] Lustre: 348818:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11394.404428] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11394.410516] Lustre: Skipped 1 previous similar message [11394.490578] Lustre: lustre-MDT0001: new disk, initializing [11394.657407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11394.668443] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11399.347864] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [11408.523910] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11414.082715] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11420.009716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11420.241814] Lustre: lustre-OST0000: new disk, initializing [11420.245599] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11420.251569] Lustre: 350784:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11422.175991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11422.186215] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11422.305142] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11426.246679] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [11435.618697] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11440.510128] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11440.694804] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11445.510648] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11445.756592] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11447.777696] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11447.785159] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11447.787411] LustreError: Skipped 1 previous similar message [11447.788425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11447.788430] Lustre: Skipped 2 previous similar messages [11447.825147] Lustre: Skipped 1 previous similar message [11452.903586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11452.910297] Lustre: Skipped 2 previous similar messages [11453.811499] Lustre: server umount lustre-OST0000 complete [11458.018815] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11458.032343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11458.038559] Lustre: Skipped 1 previous similar message [11468.261747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11468.269509] Lustre: Skipped 3 previous similar messages [11471.328134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11471.510249] Lustre: server umount lustre-MDT0000 complete [11473.378148] LustreError: 348830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11473.405528] LustreError: 348830:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11474.907294] LustreError: 349621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787591033 with bad export cookie 12714713649898019269 [11474.917232] LustreError: 349621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11474.917266] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11475.206166] Lustre: server umount lustre-MDT0001 complete [11483.002152] XFS (loop0): Unmounting Filesystem [11488.054661] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 13:04:05 (1787591045) [11494.155243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11494.602363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11494.610799] Lustre: Skipped 2 previous similar messages [11498.241865] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [11506.560369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11511.183269] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [11518.254620] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11523.077273] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11528.773115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11534.411878] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [11540.744405] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11549.675945] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11549.676606] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11549.684559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11549.693292] Lustre: Skipped 2 previous similar messages [11549.703879] Lustre: Skipped 1 previous similar message [11553.987935] Lustre: server umount lustre-OST0000 complete [11558.372345] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11563.527021] Lustre: server umount lustre-MDT0000 complete [11565.025978] LustreError: 354373:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11565.037201] LustreError: 354373:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11566.696281] LustreError: 355934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787591125 with bad export cookie 12714713649898020151 [11566.698572] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11566.708824] LustreError: 355934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11567.083964] Lustre: server umount lustre-MDT0001 complete [11577.105390] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [11580.349524] Key type lgssc unregistered [11580.711574] LNet: 356978:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11580.721738] LNetError: 356978:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11581.799766] LNet: Removed LNI 192.168.206.154@tcp [11582.912253] Key type .llcrypt unregistered [11582.915262] Key type ._llcrypt unregistered [11616.271270] Key type ._llcrypt registered [11616.273400] Key type .llcrypt registered [11616.382803] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [11617.220864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11617.325578] alg: No test for adler32 (adler32-zlib) [11618.413265] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [11618.578772] LNet: Added LNI 192.168.206.154@tcp [8/256/0/180] [11620.210085] Key type lgssc registered [11621.424166] Lustre: Echo OBD driver; http://www.lustre.org/ [11629.461317] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11629.501497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11630.940242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11635.608606] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [11640.293276] LustreError: 358550:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11640.319464] LustreError: 358550:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11644.964646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11645.483986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11650.668848] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [11658.795352] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11665.278457] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11672.020029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11672.484617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11678.920460] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [11686.528902] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11698.694055] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.54@tcp (stopping) [11701.222285] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11701.235291] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11701.242705] Lustre: Skipped 1 previous similar message [11703.250395] Lustre: server umount lustre-MDT0001 complete [11703.777362] LustreError: 359900:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11706.342176] LustreError: 358545:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11706.376330] LustreError: 358545:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11708.898343] LustreError: 359900:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11714.014956] LustreError: 358544:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11714.051374] LustreError: 358544:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11714.906393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11715.025453] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11715.038365] Lustre: Found index 1 for lustre-MDT0001, updating log [11715.092460] Lustre: 358538:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11715.411031] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11720.262229] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [11720.715343] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11722.752570] LustreError: 362119:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [11722.760038] LustreError: 362119:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [11722.771828] LustreError: 358541:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [11722.843069] LustreError: 362123:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.154@tcp: cfg command failed: rc = -17 [11722.863399] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [11722.863399] [11729.404481] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.54@tcp (stopping) [11730.914811] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11730.918083] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11730.945460] Lustre: Skipped 2 previous similar messages [11734.505259] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.54@tcp (stopping) [11734.524479] Lustre: Skipped 2 previous similar messages [11734.592331] Lustre: server umount lustre-MDT0001 complete [11736.036511] LustreError: 358549:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11736.051923] LustreError: 358549:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11743.732253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11743.818671] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11743.823678] Lustre: Found index 1 for lustre-MDT0001, updating log [11743.857460] Lustre: 358539:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11744.249555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11748.578686] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [11749.369261] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11749.884385] LustreError: 363144:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [11749.891611] LustreError: 363144:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [11749.900289] LustreError: 358541:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [11749.907686] LustreError: 358541:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [11749.961726] LustreError: 363151:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [11749.966753] LustreError: 363151:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [11749.975331] LustreError: 363151:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.154@tcp: cfg command failed: rc = -17 [11749.993923] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.206.154@tcp [11749.993923] [12056.547898] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12056.558099] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12056.574880] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12056.583107] Lustre: Skipped 1 previous similar message [12060.856713] Lustre: server umount lustre-MDT0001 complete [12061.666559] LustreError: 358550:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12061.684422] LustreError: 358550:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12070.338980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12070.490161] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12070.501524] Lustre: Found index 1 for lustre-MDT0001, updating log [12070.589948] Lustre: 358537:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12070.914753] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12075.451639] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [12076.028331] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12079.099182] LustreError: 372300:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12079.108686] LustreError: 372300:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12079.115564] LustreError: 358541:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12079.121728] LustreError: 358541:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12079.217370] LustreError: 372303:0:(lod_lov.c:208:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [12079.233676] LustreError: 372303:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.154@tcp: cfg command failed: rc = -114 [12079.246300] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [12079.246300] [12398.582285] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12398.591939] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12398.596784] Lustre: Skipped 1 previous similar message [12398.597486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12398.597491] Lustre: Skipped 1 previous similar message [12412.384117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12412.600463] Lustre: server umount lustre-MDT0000 complete [12413.926432] LustreError: 358549:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12413.944979] LustreError: 358549:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [12415.787567] LustreError: 358529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787591974 with bad export cookie 6756079228476318778 [12415.790384] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12415.799243] LustreError: 358529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12419.047506] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12419.060157] Lustre: Skipped 2 previous similar messages [12419.066078] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12419.069875] Lustre: Skipped 8 previous similar messages [12430.304637] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12430.576780] Lustre: server umount lustre-MDT0001 complete [12433.694976] Lustre: server umount lustre-OST0000 complete [12440.734706] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [12449.354810] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [12484.539652] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 13:20:41 (1787592041) [12490.432746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12490.656942] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12490.680928] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12490.766695] Lustre: lustre-MDT0000: new disk, initializing [12490.842426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12490.856510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12495.457953] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [12505.092076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12505.243221] Lustre: 383236:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12505.314687] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12505.320979] Lustre: Skipped 1 previous similar message [12505.387710] Lustre: lustre-MDT0001: new disk, initializing [12505.479569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12505.511984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12505.527808] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12509.450765] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [12518.703172] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12524.424274] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12531.933436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12532.316543] Lustre: lustre-OST0000: new disk, initializing [12532.321554] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12532.333400] Lustre: 385204:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12532.439623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12537.895588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12537.919359] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12538.033093] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12539.811042] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_default_debug -1 all [12552.273350] Lustre: DEBUG MARKER: oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12560.874848] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12560.885314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12560.896887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12560.899450] Lustre: Skipped 3 previous similar messages [12561.460585] Lustre: server umount lustre-MDT0000 complete [12563.945277] LustreError: 385925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12563.966155] LustreError: 385925:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [12565.035193] LustreError: 383228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787592123 with bad export cookie 6756079228476320731 [12565.037593] LustreError: MGC192.168.206.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12565.054365] LustreError: 383228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12565.066206] LustreError: 384042:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [12565.376487] Lustre: server umount lustre-MDT0001 complete [12580.049577] Lustre: server umount lustre-OST0000 complete [12609.364473] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing set_hostid [12619.656903] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing load_modules_local [12646.595777] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12341 sec ========= 13:23:24 (1787592204) [12648.212913] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:23:25 (1787592205) === [12650.988577] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:23:28 (1787592208) === [12688.194489] Lustre: DEBUG MARKER: oleg654-server.virtnet: executing unload_modules_local [12690.837123] Key type lgssc unregistered [12691.101582] LNet: 391572:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12691.107198] LNetError: 391572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12691.126499] LNet: Removed LNI 192.168.206.154@tcp [12692.208851] Key type .llcrypt unregistered [12692.212545] Key type ._llcrypt unregistered